00010000:00010000:3.0:1787827202.339795:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7084800/0x268e1a602cc0783 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2da0 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.339815:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7084800/0x268e1a602cc0783 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2da0 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.339852:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7085000/0x268e1a602cc078a lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2dbc expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.339906:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x56:0x0](ffff8ab3d1d16c08) 00000080:00200000:3.0:1787827202.339911:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x56:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.339931:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x56:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.339942:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x56:0x0](ffff8ab3d1d16c08) for lock 0x268e1a602cc078a 00000080:00010000:3.0:1787827202.339950:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc078a from it ffff8ab3f0477590 00010000:00010000:3.0:1787827202.339960:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7085000/0x268e1a602cc078a lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2dbc expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.339989:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7085000/0x268e1a602cc078a lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2dbc expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.340060:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7084200/0x268e1a602cc0791 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2dd8 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.340092:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x21:0x0](ffff8ab3d1e6b248) 00000080:00200000:3.0:1787827202.340282:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x21:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.340448:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x21:0x0](ffff8ab3d1e6b248) for lock 0x268e1a602cc0791 00000080:00010000:3.0:1787827202.340461:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0791 from it ffff8ab3f0474590 00010000:00010000:3.0:1787827202.340467:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7084200/0x268e1a602cc0791 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2dd8 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.340579:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7084200/0x268e1a602cc0791 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2dd8 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.340690:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7084000/0x268e1a602cc0798 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x58:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2e09 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.340730:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x58:0x0](ffff8ab3d1e68908) 00000080:00200000:3.0:1787827202.340740:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x58:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.340758:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x58:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.340766:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x58:0x0](ffff8ab3d1e68908) for lock 0x268e1a602cc0798 00000080:00010000:3.0:1787827202.340778:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0798 from it ffff8ab3f0476590 00010000:00010000:3.0:1787827202.340786:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7084000/0x268e1a602cc0798 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x58:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2e09 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.340812:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7084000/0x268e1a602cc0798 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x58:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2e09 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.340848:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7085a00/0x268e1a602cc079f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2e2c expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.340882:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x46:0x0](ffff8ab3d1e6db88) 00000080:00200000:3.0:1787827202.340940:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x46:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.340945:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x46:0x0](ffff8ab3d1e6db88) for lock 0x268e1a602cc079f 00000080:00010000:3.0:1787827202.340951:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc079f from it ffff8ab3f0475990 00010000:00010000:3.0:1787827202.341036:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7085a00/0x268e1a602cc079f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2e2c expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.341267:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7085a00/0x268e1a602cc079f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2e2c expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.341390:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7085800/0x268e1a602cc07a6 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2e5d expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.341424:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x6e:0x0](ffff8ab3d1e69148) 00000080:00200000:3.0:1787827202.341434:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x6e:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.341454:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x6e:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.341462:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6e:0x0](ffff8ab3d1e69148) for lock 0x268e1a602cc07a6 00000080:00010000:3.0:1787827202.341468:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc07a6 from it ffff8ab3f0476990 00010000:00010000:3.0:1787827202.341474:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7085800/0x268e1a602cc07a6 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2e5d expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.341500:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c7085800/0x268e1a602cc07a6 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2e5d expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.341545:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7800/0x268e1a602cc07ad lrc: 3/1,0 mode: PR/PR res: [0x200000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2e80 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.341754:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0xd:0x0](ffff8ab3d1e69988) 00000080:00200000:3.0:1787827202.341800:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0xd:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.341809:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0xd:0x0](ffff8ab3d1e69988) for lock 0x268e1a602cc07ad 00000080:00010000:3.0:1787827202.341821:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc07ad from it ffff8ab3c3767990 00010000:00010000:3.0:1787827202.341828:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7800/0x268e1a602cc07ad lrc: 3/1,0 mode: PR/PR res: [0x200000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2e80 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.342016:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7800/0x268e1a602cc07ad lrc: 2/0,0 mode: PR/PR res: [0x200000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2e80 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.342065:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6000/0x268e1a602cc07b4 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2e9c expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.342170:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0xe:0x0](ffff8ab3d1e6aa08) 00000080:00200000:3.0:1787827202.342218:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0xe:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.342224:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0xe:0x0](ffff8ab3d1e6aa08) for lock 0x268e1a602cc07b4 00000080:00010000:3.0:1787827202.342236:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc07b4 from it ffff8ab3c3766190 00010000:00010000:3.0:1787827202.342308:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6000/0x268e1a602cc07b4 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2e9c expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.342379:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6000/0x268e1a602cc07b4 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2e9c expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.342472:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6600/0x268e1a602cc07bb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2eb8 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.342513:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x49:0x0](ffff8ab3d1e6e3c8) 00000080:00200000:3.0:1787827202.342554:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x49:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.342561:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x49:0x0](ffff8ab3d1e6e3c8) for lock 0x268e1a602cc07bb 00000080:00010000:3.0:1787827202.342571:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc07bb from it ffff8ab3c3765190 00010000:00010000:3.0:1787827202.342576:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6600/0x268e1a602cc07bb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2eb8 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.342597:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6600/0x268e1a602cc07bb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2eb8 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.342707:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7200/0x268e1a602cc07c2 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2ed4 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.342749:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x25:0x0](ffff8ab3d1e6cb08) 00000080:00200000:3.0:1787827202.342787:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x25:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.342793:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x25:0x0](ffff8ab3d1e6cb08) for lock 0x268e1a602cc07c2 00000080:00010000:3.0:1787827202.342809:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc07c2 from it ffff8ab3c3764d90 00010000:00010000:3.0:1787827202.342813:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7200/0x268e1a602cc07c2 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2ed4 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.342824:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7200/0x268e1a602cc07c2 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2ed4 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.342855:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6800/0x268e1a602cc07c9 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2ef0 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.342874:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x98:0x0](ffff8ab3d1e680c8) 00000080:00200000:3.0:1787827202.342879:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x98:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.342930:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x98:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.343032:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x98:0x0](ffff8ab3d1e680c8) for lock 0x268e1a602cc07c9 00000080:00010000:3.0:1787827202.343060:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc07c9 from it ffff8ab3c3764990 00010000:00010000:3.0:1787827202.343068:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6800/0x268e1a602cc07c9 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2ef0 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.343099:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6800/0x268e1a602cc07c9 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2ef0 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.343152:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6a00/0x268e1a602cc07d0 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2f2f expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.343183:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x75:0x0](ffff8ab3d1e6f448) 00000080:00200000:3.0:1787827202.343189:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x75:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.343207:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x75:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.343214:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x75:0x0](ffff8ab3d1e6f448) for lock 0x268e1a602cc07d0 00000080:00010000:3.0:1787827202.343238:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc07d0 from it ffff8ab3c3767d90 00010000:00010000:3.0:1787827202.343245:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6a00/0x268e1a602cc07d0 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2f2f expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.343263:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6a00/0x268e1a602cc07d0 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2f2f expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.343285:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6c00/0x268e1a602cc07d7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2f52 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.343300:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x23:0x0](ffff8ab3d1e6d348) 00000080:00200000:3.0:1787827202.343334:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x23:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.343339:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x23:0x0](ffff8ab3d1e6d348) for lock 0x268e1a602cc07d7 00000080:00010000:3.0:1787827202.343344:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc07d7 from it ffff8ab3c3765590 00010000:00010000:3.0:1787827202.343347:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6c00/0x268e1a602cc07d7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2f52 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.343463:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6c00/0x268e1a602cc07d7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2f52 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.343487:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6400/0x268e1a602cc07de lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2f6e expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.343516:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x1f:0x0](ffff8ab3d1e6ba88) 00000080:00200000:3.0:1787827202.343556:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x1f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.343563:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1f:0x0](ffff8ab3d1e6ba88) for lock 0x268e1a602cc07de 00000080:00010000:3.0:1787827202.343569:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc07de from it ffff8ab3c3767190 00010000:00010000:3.0:1787827202.343575:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6400/0x268e1a602cc07de lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2f6e expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.343600:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6400/0x268e1a602cc07de lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2f6e expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.343837:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6200/0x268e1a602cc07e5 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x8d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2f8a expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.343859:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x8d:0x0](ffff8ab3d1e6ec08) 00000080:00200000:3.0:1787827202.343867:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x8d:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.344014:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x8d:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.344025:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x8d:0x0](ffff8ab3d1e6ec08) for lock 0x268e1a602cc07e5 00000080:00010000:3.0:1787827202.344035:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc07e5 from it ffff8ab3c3765d90 00010000:00010000:3.0:1787827202.344042:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6200/0x268e1a602cc07e5 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x8d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2f8a expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.344072:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6200/0x268e1a602cc07e5 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x8d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2f8a expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.344105:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6e00/0x268e1a602cc07ec lrc: 3/1,0 mode: PR/PR res: [0x200000402:0xac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2fc2 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.344151:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0xac:0x0](ffff8ab3d1e6c2c8) 00000080:00200000:3.0:1787827202.344157:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0xac:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.344173:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0xac:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.344177:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0xac:0x0](ffff8ab3d1e6c2c8) for lock 0x268e1a602cc07ec 00000080:00010000:3.0:1787827202.344194:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc07ec from it ffff8ab3c3764190 00010000:00010000:3.0:1787827202.344198:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6e00/0x268e1a602cc07ec lrc: 3/1,0 mode: PR/PR res: [0x200000402:0xac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2fc2 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.344206:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f6e00/0x268e1a602cc07ec lrc: 2/0,0 mode: PR/PR res: [0x200000402:0xac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2fc2 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.344227:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7c00/0x268e1a602cc07f3 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0xa3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2fde expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.344239:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0xa3:0x0](ffff8ab3d1e6a1c8) 00000080:00200000:3.0:1787827202.344243:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0xa3:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.344253:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0xa3:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.344256:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0xa3:0x0](ffff8ab3d1e6a1c8) for lock 0x268e1a602cc07f3 00000080:00010000:3.0:1787827202.344260:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc07f3 from it ffff8ab3c3766d90 00010000:00010000:3.0:1787827202.344263:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7c00/0x268e1a602cc07f3 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0xa3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2fde expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.344271:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7c00/0x268e1a602cc07f3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0xa3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2fde expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.344398:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7600/0x268e1a602cc07fa lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d2ffa expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.344418:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x9f:0x0](ffff8ab3f13963c8) 00000080:00200000:3.0:1787827202.344421:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x9f:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.344437:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x9f:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.344441:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x9f:0x0](ffff8ab3f13963c8) for lock 0x268e1a602cc07fa 00000080:00010000:3.0:1787827202.344458:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc07fa from it ffff8ab3c3767590 00010000:00010000:3.0:1787827202.344463:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7600/0x268e1a602cc07fa lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d2ffa expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.344641:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7600/0x268e1a602cc07fa lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d2ffa expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.344690:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7e00/0x268e1a602cc0801 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d3016 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.344717:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0xa6:0x0](ffff8ab3f1391148) 00000080:00200000:3.0:1787827202.344723:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0xa6:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.344753:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0xa6:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.344759:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0xa6:0x0](ffff8ab3f1391148) for lock 0x268e1a602cc0801 00000080:00010000:3.0:1787827202.344772:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0801 from it ffff8ab3c3764590 00010000:00010000:3.0:1787827202.344781:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7e00/0x268e1a602cc0801 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d3016 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.344794:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7e00/0x268e1a602cc0801 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d3016 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.344821:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7400/0x268e1a602cc0808 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d3032 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.344855:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0xc:0x0](ffff8ab3d1f8cb08) 00000080:00200000:3.0:1787827202.345376:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0xc:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.345386:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0xc:0x0](ffff8ab3d1f8cb08) for lock 0x268e1a602cc0808 00000080:00010000:3.0:1787827202.345399:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0808 from it ffff8ab3c3766590 00010000:00010000:3.0:1787827202.345410:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7400/0x268e1a602cc0808 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d3032 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.345437:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7400/0x268e1a602cc0808 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d3032 expref: -99 pid: 48407 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.345476:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7000/0x268e1a602cc080f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d3063 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.345507:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x90:0x0](ffff8ab3d1f8f448) 00000080:00200000:3.0:1787827202.345516:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x90:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.345533:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x90:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.345542:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x90:0x0](ffff8ab3d1f8f448) for lock 0x268e1a602cc080f 00000080:00010000:3.0:1787827202.345551:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc080f from it ffff8ab3c3765990 00010000:00010000:3.0:1787827202.345558:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7000/0x268e1a602cc080f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d3063 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.345602:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7000/0x268e1a602cc080f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d3063 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.345638:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7a00/0x268e1a602cc0816 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d3086 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.345670:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x6c:0x0](ffff8ab3d1f8db88) 00000080:00200000:3.0:1787827202.345677:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x6c:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.345691:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x6c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.345695:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6c:0x0](ffff8ab3d1f8db88) for lock 0x268e1a602cc0816 00000080:00010000:3.0:1787827202.345700:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0816 from it ffff8ab3c3766990 00010000:00010000:3.0:1787827202.345707:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7a00/0x268e1a602cc0816 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d3086 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.345729:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c49f7a00/0x268e1a602cc0816 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d3086 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.345746:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854000/0x268e1a602cc081d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d30a2 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.345754:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x55:0x0](ffff8ab3d1f8ec08) 00000080:00200000:3.0:1787827202.345760:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x55:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.345770:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x55:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.345777:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x55:0x0](ffff8ab3d1f8ec08) for lock 0x268e1a602cc081d 00000080:00010000:3.0:1787827202.345782:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc081d from it ffff8ab3d0b01590 00010000:00010000:3.0:1787827202.345787:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854000/0x268e1a602cc081d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d30a2 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.345802:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854000/0x268e1a602cc081d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d30a2 expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.345820:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0855000/0x268e1a602cc0824 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0xae:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d30be expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.345842:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0xae:0x0](ffff8ab3d1f88908) 00000080:00200000:3.0:1787827202.345849:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0xae:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.345867:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0xae:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.345877:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0xae:0x0](ffff8ab3d1f88908) for lock 0x268e1a602cc0824 00000080:00010000:3.0:1787827202.345883:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0824 from it ffff8ab3d0b03190 00010000:00010000:3.0:1787827202.345910:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0855000/0x268e1a602cc0824 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0xae:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d30be expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.345953:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0855000/0x268e1a602cc0824 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0xae:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d30be expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.345978:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854400/0x268e1a602cc082b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d30da expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.345990:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x68:0x0](ffff8ab3d1f8d348) 00000080:00200000:3.0:1787827202.345994:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x68:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.346003:0:14753:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x68:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.346005:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x68:0x0](ffff8ab3d1f8d348) for lock 0x268e1a602cc082b 00000080:00010000:3.0:1787827202.346015:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc082b from it ffff8ab3d0b01d90 00010000:00010000:3.0:1787827202.346017:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854400/0x268e1a602cc082b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d30da expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.346023:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854400/0x268e1a602cc082b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d30da expref: -99 pid: 48407 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.346603:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f454355a/0x268e1a602cc09f9 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48408 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.346650:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f454355a/0x268e1a602cc09f9 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d3499 expref: -99 pid: 48408 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.346662:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f454355a/0x268e1a602cc09f9 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d3499 expref: -99 pid: 48408 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.346670:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f454355a/0x268e1a602cc09f9 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d3499 expref: -99 pid: 48408 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827202.346678:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 9198, setting rss=9198; kms=9198, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f454355a/0x268e1a602cc09f9 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d3499 expref: -99 pid: 48408 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.346687:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f454355a/0x268e1a602cc09f9 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d3499 expref: -99 pid: 48408 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.346695:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f454355a/0x268e1a602cc09f9 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d3499 expref: -99 pid: 48408 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.346700:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f454355a/0x268e1a602cc09f9 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d3499 expref: -99 pid: 48408 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.346707:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f454355a/0x268e1a602cc09f9 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d3499 expref: -99 pid: 48408 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827202.346717:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0733480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280262400:192.168.203.124@tcp:101:ll_agl.0 00000100:00100000:3.0:1787827202.839411:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0530000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280336000:192.168.203.124@tcp:63:ll_sa.0 00010000:00010000:3.0:1787827202.844007:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8600/0x268e1a602cc0d4f lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a01 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844023:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8600/0x268e1a602cc0d4f lrc: 4/1,0 mode: --/PR res: [0x240000402:0x9f:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a01 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844030:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8600/0x268e1a602cc0d4f lrc: 4/1,0 mode: --/PR res: [0x240000402:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a01 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844037:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8600/0x268e1a602cc0d4f lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a01 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844046:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8600/0x268e1a602cc0d4f lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a01 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844053:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8600/0x268e1a602cc0d4f lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a01 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844058:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8600/0x268e1a602cc0d4f lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a01 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844067:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8600/0x268e1a602cc0d4f lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a01 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.844075:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0530000 x1874671280336000/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 3520/3648 e 0 to 0 dl 1787827218 ref 1 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.844083:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d4f from it ffff8ab3c5821990 00010000:00010000:3.0:1787827202.844085:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8600/0x268e1a602cc0d4f lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a01 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844091:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8600/0x268e1a602cc0d4f lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a01 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.844121:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9400/0x268e1a602cc0d56 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a1d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844130:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9400/0x268e1a602cc0d56 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x94:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a1d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844135:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9400/0x268e1a602cc0d56 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a1d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844140:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9400/0x268e1a602cc0d56 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a1d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844145:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9400/0x268e1a602cc0d56 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a1d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844155:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9400/0x268e1a602cc0d56 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a1d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844159:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9400/0x268e1a602cc0d56 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a1d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844164:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9400/0x268e1a602cc0d56 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a1d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.844170:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0530000 x1874671280336000/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 3520/3648 e 0 to 0 dl 1787827218 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.844176:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d56 from it ffff8ab3c5821590 00010000:00010000:3.0:1787827202.844178:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9400/0x268e1a602cc0d56 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a1d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844183:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9400/0x268e1a602cc0d56 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a1d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.844192:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8200/0x268e1a602cc0d5d lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a39 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844198:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8200/0x268e1a602cc0d5d lrc: 4/1,0 mode: --/PR res: [0x240000402:0x13:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a39 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844203:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8200/0x268e1a602cc0d5d lrc: 4/1,0 mode: --/PR res: [0x240000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a39 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844211:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8200/0x268e1a602cc0d5d lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a39 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844216:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8200/0x268e1a602cc0d5d lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a39 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844220:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8200/0x268e1a602cc0d5d lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a39 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844225:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8200/0x268e1a602cc0d5d lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a39 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844229:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8200/0x268e1a602cc0d5d lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a39 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.844234:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0530000 x1874671280336000/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 3520/3648 e 0 to 0 dl 1787827218 ref 3 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.844239:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8200/0x268e1a602cc0d5d lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a39 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.844245:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d5d from it ffff8ab3c5821d90 00010000:00010000:3.0:1787827202.844247:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8200/0x268e1a602cc0d5d lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a39 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844251:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8200/0x268e1a602cc0d5d lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a39 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.844262:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8e00/0x268e1a602cc0d64 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a55 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844268:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8e00/0x268e1a602cc0d64 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xf:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a55 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844273:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8e00/0x268e1a602cc0d64 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a55 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844277:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8e00/0x268e1a602cc0d64 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a55 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844282:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8e00/0x268e1a602cc0d64 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a55 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844286:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8e00/0x268e1a602cc0d64 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a55 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844290:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8e00/0x268e1a602cc0d64 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a55 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844295:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8e00/0x268e1a602cc0d64 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a55 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.844300:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0530000 x1874671280336000/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 3520/3648 e 0 to 0 dl 1787827218 ref 4 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.844305:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8e00/0x268e1a602cc0d64 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a55 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.844310:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d64 from it ffff8ab3c5822990 00010000:00010000:3.0:1787827202.844312:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8e00/0x268e1a602cc0d64 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a55 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844317:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8e00/0x268e1a602cc0d64 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a55 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.844324:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8c00/0x268e1a602cc0d72 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a71 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844333:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8c00/0x268e1a602cc0d72 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x41:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a71 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844340:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8c00/0x268e1a602cc0d72 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a71 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844348:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8c00/0x268e1a602cc0d72 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a71 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844356:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8c00/0x268e1a602cc0d72 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a71 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844366:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8c00/0x268e1a602cc0d72 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a71 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844374:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8c00/0x268e1a602cc0d72 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a71 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844387:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8c00/0x268e1a602cc0d72 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a71 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.844393:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0530000 x1874671280336000/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 3520/3648 e 0 to 0 dl 1787827218 ref 5 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.844399:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8c00/0x268e1a602cc0d72 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a71 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.844405:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d72 from it ffff8ab3d0982990 00010000:00010000:3.0:1787827202.844407:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8c00/0x268e1a602cc0d72 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a71 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844412:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8c00/0x268e1a602cc0d72 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a71 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.844422:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9800/0x268e1a602cc0d79 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a8d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844429:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9800/0x268e1a602cc0d79 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x48:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a8d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844433:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9800/0x268e1a602cc0d79 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a8d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844438:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9800/0x268e1a602cc0d79 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a8d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844447:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9800/0x268e1a602cc0d79 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a8d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844452:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9800/0x268e1a602cc0d79 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a8d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844456:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9800/0x268e1a602cc0d79 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a8d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844460:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9800/0x268e1a602cc0d79 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a8d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.844465:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0530000 x1874671280336000/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 3520/3648 e 0 to 0 dl 1787827218 ref 6 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.844470:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9800/0x268e1a602cc0d79 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a8d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.844475:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d79 from it ffff8ab3d0983990 00010000:00010000:3.0:1787827202.844545:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9800/0x268e1a602cc0d79 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a8d expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844550:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9800/0x268e1a602cc0d79 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5a8d expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.844561:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9c00/0x268e1a602cc0d80 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5aa9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844573:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9c00/0x268e1a602cc0d80 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x97:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5aa9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844579:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9c00/0x268e1a602cc0d80 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5aa9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844584:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9c00/0x268e1a602cc0d80 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5aa9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844593:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9c00/0x268e1a602cc0d80 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5aa9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844604:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9c00/0x268e1a602cc0d80 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5aa9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844613:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9c00/0x268e1a602cc0d80 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5aa9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844621:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9c00/0x268e1a602cc0d80 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5aa9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.844633:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0530000 x1874671280336000/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 3520/3648 e 0 to 0 dl 1787827218 ref 7 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.844642:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d80 from it ffff8ab3d0982190 00010000:00010000:3.0:1787827202.844646:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9c00/0x268e1a602cc0d80 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5aa9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844659:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9c00/0x268e1a602cc0d80 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5aa9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.844675:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc0d87 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ac5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844689:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc0d87 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x3d:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ac5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844695:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc0d87 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ac5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844701:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc0d87 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ac5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844709:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc0d87 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ac5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844722:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc0d87 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ac5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844727:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc0d87 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ac5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844732:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc0d87 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ac5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.844742:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0530000 x1874671280336000/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 3520/3648 e 0 to 0 dl 1787827218 ref 8 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.844747:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc0d87 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ac5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.844753:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d87 from it ffff8ab3d0981190 00010000:00010000:3.0:1787827202.844755:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc0d87 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ac5 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.844761:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc0d87 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ac5 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827202.844774:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0530000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280336000:192.168.203.124@tcp:63:ll_sa.0 00010000:00010000:3.0:1787827202.844798:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8600/0x268e1a602cc0d4f lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a01 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.844814:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x9f:0x0](ffff8ab3d1f89148) 00000080:00200000:3.0:1787827202.844819:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x9f:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.844842:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x9f:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.844846:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x9f:0x0](ffff8ab3d1f89148) for lock 0x268e1a602cc0d4f 00000080:00010000:3.0:1787827202.844860:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d4f from it ffff8ab3c5821990 00010000:00010000:3.0:1787827202.844862:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8600/0x268e1a602cc0d4f lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5a01 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.844868:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8600/0x268e1a602cc0d4f lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5a01 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.844911:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9400/0x268e1a602cc0d56 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a1d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.844922:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x94:0x0](ffff8ab3d1f8aa08) 00000080:00200000:3.0:1787827202.844926:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x94:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.844937:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x94:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.844943:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x94:0x0](ffff8ab3d1f8aa08) for lock 0x268e1a602cc0d56 00000080:00010000:3.0:1787827202.844953:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d56 from it ffff8ab3c5821590 00010000:00010000:3.0:1787827202.844958:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9400/0x268e1a602cc0d56 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5a1d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.844966:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9400/0x268e1a602cc0d56 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5a1d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.844989:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8200/0x268e1a602cc0d5d lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a39 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.845002:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x13:0x0](ffff8ab3d1f880c8) 00000080:00200000:3.0:1787827202.845039:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x13:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.845042:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x13:0x0](ffff8ab3d1f880c8) for lock 0x268e1a602cc0d5d 00000080:00010000:3.0:1787827202.845051:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d5d from it ffff8ab3c5821d90 00010000:00010000:3.0:1787827202.845054:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8200/0x268e1a602cc0d5d lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5a39 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.845061:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8200/0x268e1a602cc0d5d lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5a39 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.845080:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8e00/0x268e1a602cc0d64 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a55 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.845093:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0xf:0x0](ffff8ab3d1f8b248) 00000080:00200000:3.0:1787827202.845140:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0xf:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.845145:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0xf:0x0](ffff8ab3d1f8b248) for lock 0x268e1a602cc0d64 00000080:00010000:3.0:1787827202.845150:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d64 from it ffff8ab3c5822990 00010000:00010000:3.0:1787827202.845157:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8e00/0x268e1a602cc0d64 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5a55 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.845168:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8e00/0x268e1a602cc0d64 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5a55 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.845187:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8c00/0x268e1a602cc0d72 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a71 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.845198:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x41:0x0](ffff8ab3d1f89988) 00000080:00200000:3.0:1787827202.845221:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x41:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.845230:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x41:0x0](ffff8ab3d1f89988) for lock 0x268e1a602cc0d72 00000080:00010000:3.0:1787827202.845237:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d72 from it ffff8ab3d0982990 00010000:00010000:3.0:1787827202.845243:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8c00/0x268e1a602cc0d72 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5a71 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.845252:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b8c00/0x268e1a602cc0d72 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5a71 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.845278:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9800/0x268e1a602cc0d79 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5a8d expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.845290:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x48:0x0](ffff8ab3d1f8e3c8) 00000080:00200000:3.0:1787827202.845326:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x48:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.845336:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x48:0x0](ffff8ab3d1f8e3c8) for lock 0x268e1a602cc0d79 00000080:00010000:3.0:1787827202.845349:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d79 from it ffff8ab3d0983990 00010000:00010000:3.0:1787827202.845358:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9800/0x268e1a602cc0d79 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5a8d expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.845378:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9800/0x268e1a602cc0d79 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5a8d expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.845412:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9c00/0x268e1a602cc0d80 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5aa9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.845500:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x97:0x0](ffff8ab3d1f8a1c8) 00000080:00200000:3.0:1787827202.845506:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x97:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.845522:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x97:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.845532:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x97:0x0](ffff8ab3d1f8a1c8) for lock 0x268e1a602cc0d80 00000080:00010000:3.0:1787827202.845540:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d80 from it ffff8ab3d0982190 00010000:00010000:3.0:1787827202.845547:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9c00/0x268e1a602cc0d80 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5aa9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.845567:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c29b9c00/0x268e1a602cc0d80 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5aa9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.845612:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc0d87 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ac5 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.845632:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x3d:0x0](ffff8ab3d1f8ba88) 00000080:00200000:3.0:1787827202.845748:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x3d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.845754:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x3d:0x0](ffff8ab3d1f8ba88) for lock 0x268e1a602cc0d87 00000080:00010000:3.0:1787827202.845768:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0d87 from it ffff8ab3d0981190 00010000:00010000:3.0:1787827202.845804:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc0d87 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5ac5 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.845813:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc0d87 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5ac5 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.846525:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1702dbd/0x268e1a602cc0d9c lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x21:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.846535:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1702dbd/0x268e1a602cc0d9c lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x21:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ada expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.846543:0:10957:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1702dbd/0x268e1a602cc0d9c lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x21:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ada expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.846550:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1702dbd/0x268e1a602cc0d9c lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x21:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ada expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827202.846559:0:10957:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 159678, setting rss=159678; kms=159678, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1702dbd/0x268e1a602cc0d9c lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x21:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5ada expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.846569:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1702dbd/0x268e1a602cc0d9c lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x21:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5ada expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.846575:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1702dbd/0x268e1a602cc0d9c lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x21:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5ada expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.846580:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1702dbd/0x268e1a602cc0d9c lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x21:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5ada expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.846587:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1702dbd/0x268e1a602cc0d9c lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x21:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5ada expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827202.846687:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04c61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280336512:192.168.203.124@tcp:101:ll_agl.0 00010000:00010000:3.0:1787827202.846713:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000003491c5f8/0x268e1a602cc0d95 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.846739:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000003491c5f8/0x268e1a602cc0d95 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ad3 expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.846803:0:10957:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000003491c5f8/0x268e1a602cc0d95 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ad3 expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.846830:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000003491c5f8/0x268e1a602cc0d95 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ad3 expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827202.846857:0:10957:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 704, setting rss=704; kms=704, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000003491c5f8/0x268e1a602cc0d95 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5ad3 expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.846878:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000003491c5f8/0x268e1a602cc0d95 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5ad3 expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.846915:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000003491c5f8/0x268e1a602cc0d95 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5ad3 expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.846927:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000003491c5f8/0x268e1a602cc0d95 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5ad3 expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.846934:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000003491c5f8/0x268e1a602cc0d95 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5ad3 expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827202.846947:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04c4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280336384:192.168.203.124@tcp:101:ll_agl.0 00000100:00100000:3.0:1787827202.860138:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0532940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280339456:192.168.203.124@tcp:63:ll_sa.0 00000100:00100000:3.0:1787827202.862294:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0531e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280339840:192.168.203.124@tcp:63:ll_sa.0 00000100:00100000:3.0:1787827202.865409:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0b60000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280340480:192.168.203.124@tcp:101:ll_agl.0 00010000:00010000:3.0:1787827202.873718:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7600/0x268e1a602cc0e0c lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 50 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5cb6 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873736:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7600/0x268e1a602cc0e0c lrc: 4/1,0 mode: --/PR res: [0x240000402:0x7b:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5cb6 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873741:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7600/0x268e1a602cc0e0c lrc: 4/1,0 mode: --/PR res: [0x240000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5cb6 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873748:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7600/0x268e1a602cc0e0c lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5cb6 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873755:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7600/0x268e1a602cc0e0c lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5cb6 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873761:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7600/0x268e1a602cc0e0c lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5cb6 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873766:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7600/0x268e1a602cc0e0c lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5cb6 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873771:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7600/0x268e1a602cc0e0c lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5cb6 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.873778:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 1 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.873786:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e0c from it ffff8ab3d0980990 00010000:00010000:3.0:1787827202.873788:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7600/0x268e1a602cc0e0c lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5cb6 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873797:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7600/0x268e1a602cc0e0c lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5cb6 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.873813:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc0e13 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 49 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d18 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873820:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc0e13 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x8a:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d18 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873825:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc0e13 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d18 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873830:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc0e13 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d18 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873850:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc0e13 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d18 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873856:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc0e13 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d18 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873861:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc0e13 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d18 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873867:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc0e13 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d18 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.873877:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.873884:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e13 from it ffff8ab3d0980d90 00010000:00010000:3.0:1787827202.873905:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc0e13 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d18 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873911:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc0e13 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d18 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.873923:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7800/0x268e1a602cc0e1a lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 48 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d42 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873932:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7800/0x268e1a602cc0e1a lrc: 4/1,0 mode: --/PR res: [0x240000402:0x33:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d42 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873937:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7800/0x268e1a602cc0e1a lrc: 4/1,0 mode: --/PR res: [0x240000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d42 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873942:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7800/0x268e1a602cc0e1a lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d42 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873948:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7800/0x268e1a602cc0e1a lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d42 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873954:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7800/0x268e1a602cc0e1a lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d42 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873967:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7800/0x268e1a602cc0e1a lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d42 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.873976:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7800/0x268e1a602cc0e1a lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d42 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.873986:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 3 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.873998:0:10957:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7800/0x268e1a602cc0e1a lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d42 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.874009:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e1a from it ffff8ab3d0980190 00010000:00010000:3.0:1787827202.874013:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7800/0x268e1a602cc0e1a lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d42 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874021:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7800/0x268e1a602cc0e1a lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d42 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.874037:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc0e21 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 47 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d5e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874129:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc0e21 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xa8:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d5e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874137:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc0e21 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d5e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874147:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc0e21 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d5e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874154:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc0e21 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d5e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874162:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc0e21 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d5e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874167:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc0e21 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d5e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874172:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc0e21 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d5e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.874181:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 4 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.874187:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e21 from it ffff8ab3d0981d90 00010000:00010000:3.0:1787827202.874190:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc0e21 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d5e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874197:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc0e21 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d5e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.874212:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6a00/0x268e1a602cc0e28 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 46 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d96 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874228:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6a00/0x268e1a602cc0e28 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x6e:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d96 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874238:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6a00/0x268e1a602cc0e28 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d96 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874248:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6a00/0x268e1a602cc0e28 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d96 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874253:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6a00/0x268e1a602cc0e28 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d96 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874269:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6a00/0x268e1a602cc0e28 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d96 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874280:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6a00/0x268e1a602cc0e28 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d96 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874286:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6a00/0x268e1a602cc0e28 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d96 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.874293:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 5 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.874298:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e28 from it ffff8ab3d0983190 00010000:00010000:3.0:1787827202.874301:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6a00/0x268e1a602cc0e28 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d96 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874310:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6a00/0x268e1a602cc0e28 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5d96 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.874325:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc0e2f lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 45 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5dce expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874333:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc0e2f lrc: 4/1,0 mode: --/PR res: [0x240000402:0x4d:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5dce expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874340:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc0e2f lrc: 4/1,0 mode: --/PR res: [0x240000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5dce expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874348:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc0e2f lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5dce expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874360:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc0e2f lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5dce expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874368:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc0e2f lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5dce expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874377:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc0e2f lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5dce expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874391:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc0e2f lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5dce expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.874415:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 6 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.874424:0:10957:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc0e2f lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5dce expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.874436:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e2f from it ffff8ab3d0982590 00010000:00010000:3.0:1787827202.874439:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc0e2f lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5dce expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874448:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc0e2f lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5dce expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.874462:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc0e36 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e22 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874474:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc0e36 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xa6:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e22 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874487:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc0e36 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e22 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874498:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc0e36 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e22 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874507:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc0e36 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e22 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874522:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc0e36 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e22 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874541:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc0e36 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e22 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874549:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc0e36 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e22 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.874560:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 7 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.874566:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e36 from it ffff8ab3d0981990 00010000:00010000:3.0:1787827202.874568:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc0e36 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e22 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874574:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc0e36 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e22 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.874587:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f400/0x268e1a602cc0e3d lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 43 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e61 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874604:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f400/0x268e1a602cc0e3d lrc: 4/1,0 mode: --/PR res: [0x240000402:0xa0:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e61 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874612:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f400/0x268e1a602cc0e3d lrc: 4/1,0 mode: --/PR res: [0x240000402:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e61 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874628:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f400/0x268e1a602cc0e3d lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e61 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874637:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f400/0x268e1a602cc0e3d lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e61 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874646:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f400/0x268e1a602cc0e3d lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e61 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874652:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f400/0x268e1a602cc0e3d lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e61 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874673:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f400/0x268e1a602cc0e3d lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e61 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.874685:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 8 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.874694:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e3d from it ffff8ab3d0bd5990 00010000:00010000:3.0:1787827202.874698:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f400/0x268e1a602cc0e3d lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e61 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874707:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f400/0x268e1a602cc0e3d lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e61 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.874778:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fc00/0x268e1a602cc0e44 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 42 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ec3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874790:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fc00/0x268e1a602cc0e44 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x9c:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ec3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874795:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fc00/0x268e1a602cc0e44 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ec3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874800:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fc00/0x268e1a602cc0e44 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ec3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874805:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fc00/0x268e1a602cc0e44 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ec3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874811:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fc00/0x268e1a602cc0e44 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ec3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874815:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fc00/0x268e1a602cc0e44 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ec3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874820:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fc00/0x268e1a602cc0e44 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ec3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.874826:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 9 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.874831:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e44 from it ffff8ab3d0bd6990 00010000:00010000:3.0:1787827202.874833:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fc00/0x268e1a602cc0e44 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ec3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874841:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fc00/0x268e1a602cc0e44 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ec3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.874850:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc0e4b lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 41 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f1e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874857:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc0e4b lrc: 4/1,0 mode: --/PR res: [0x240000402:0x43:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f1e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874861:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc0e4b lrc: 4/1,0 mode: --/PR res: [0x240000402:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f1e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874866:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc0e4b lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f1e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874871:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc0e4b lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f1e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874876:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc0e4b lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f1e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874881:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc0e4b lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f1e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874905:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc0e4b lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f1e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.874915:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 10 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.874922:0:10957:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc0e4b lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f1e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.874929:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e4b from it ffff8ab3d0bd7990 00010000:00010000:3.0:1787827202.874931:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc0e4b lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f1e expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874936:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc0e4b lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f1e expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.874945:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e800/0x268e1a602cc0e52 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 40 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f5d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874952:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e800/0x268e1a602cc0e52 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x75:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f5d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874957:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e800/0x268e1a602cc0e52 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f5d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874962:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e800/0x268e1a602cc0e52 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f5d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874967:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e800/0x268e1a602cc0e52 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f5d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874975:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e800/0x268e1a602cc0e52 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f5d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874980:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e800/0x268e1a602cc0e52 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f5d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.874984:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e800/0x268e1a602cc0e52 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f5d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.874989:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 11 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.874994:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e52 from it ffff8ab3d0bd6190 00010000:00010000:3.0:1787827202.874996:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e800/0x268e1a602cc0e52 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f5d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875000:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e800/0x268e1a602cc0e52 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f5d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.875007:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e600/0x268e1a602cc0e59 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 39 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f9c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875013:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e600/0x268e1a602cc0e59 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x29:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f9c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875018:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e600/0x268e1a602cc0e59 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f9c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875026:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e600/0x268e1a602cc0e59 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f9c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875031:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e600/0x268e1a602cc0e59 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f9c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875036:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e600/0x268e1a602cc0e59 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f9c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875040:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e600/0x268e1a602cc0e59 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f9c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875044:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e600/0x268e1a602cc0e59 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f9c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.875049:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 12 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.875054:0:10957:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e600/0x268e1a602cc0e59 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f9c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.875058:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e59 from it ffff8ab3d0bd5190 00010000:00010000:3.0:1787827202.875060:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e600/0x268e1a602cc0e59 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f9c expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875064:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e600/0x268e1a602cc0e59 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f9c expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.875075:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc0e60 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 38 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ff7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875082:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc0e60 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x19:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ff7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875086:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc0e60 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ff7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875090:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc0e60 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ff7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875095:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc0e60 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ff7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875099:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc0e60 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ff7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875104:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc0e60 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ff7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875125:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc0e60 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ff7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.875131:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 13 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.875140:0:10957:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc0e60 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ff7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.875146:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e60 from it ffff8ab3d0bd4d90 00010000:00010000:3.0:1787827202.875148:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc0e60 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ff7 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875153:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc0e60 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5ff7 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.875163:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f600/0x268e1a602cc0e67 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6044 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875170:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f600/0x268e1a602cc0e67 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x34:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6044 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875175:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f600/0x268e1a602cc0e67 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6044 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875180:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f600/0x268e1a602cc0e67 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6044 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875185:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f600/0x268e1a602cc0e67 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6044 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875189:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f600/0x268e1a602cc0e67 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6044 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875197:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f600/0x268e1a602cc0e67 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6044 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875202:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f600/0x268e1a602cc0e67 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6044 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.875207:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 14 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.875212:0:10957:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f600/0x268e1a602cc0e67 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6044 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.875216:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e67 from it ffff8ab3d0bd4990 00010000:00010000:3.0:1787827202.875218:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f600/0x268e1a602cc0e67 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6044 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875222:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f600/0x268e1a602cc0e67 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6044 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.875229:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc0e6e lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d609f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875235:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc0e6e lrc: 4/1,0 mode: --/PR res: [0x240000402:0x74:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d609f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875239:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc0e6e lrc: 4/1,0 mode: --/PR res: [0x240000402:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d609f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875247:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc0e6e lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d609f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875251:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc0e6e lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d609f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875256:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc0e6e lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d609f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875260:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc0e6e lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d609f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875265:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc0e6e lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d609f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.875269:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 15 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.875274:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e6e from it ffff8ab3d0bd7d90 00010000:00010000:3.0:1787827202.875276:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc0e6e lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d609f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875280:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc0e6e lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d609f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.875287:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc0e75 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 35 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60ec expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875296:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc0e75 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xd:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60ec expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875301:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc0e75 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60ec expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875305:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc0e75 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60ec expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875309:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc0e75 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60ec expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875314:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc0e75 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60ec expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875318:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc0e75 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60ec expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875323:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc0e75 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60ec expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.875327:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 16 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.875332:0:10957:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc0e75 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60ec expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.875340:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e75 from it ffff8ab3d0bd5590 00010000:00010000:3.0:1787827202.875342:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc0e75 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60ec expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875346:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc0e75 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60ec expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.875354:0:10957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031800/0x268e1a602cc0e7c lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 34 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d612b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875360:0:10957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031800/0x268e1a602cc0e7c lrc: 4/1,0 mode: --/PR res: [0x240000402:0x57:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d612b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875364:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031800/0x268e1a602cc0e7c lrc: 4/1,0 mode: --/PR res: [0x240000402:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d612b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875368:0:10957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031800/0x268e1a602cc0e7c lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d612b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875373:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031800/0x268e1a602cc0e7c lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d612b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875378:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031800/0x268e1a602cc0e7c lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d612b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875382:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031800/0x268e1a602cc0e7c lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d612b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875390:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031800/0x268e1a602cc0e7c lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d612b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.875395:0:10957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0532940 x1874671280339456/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 272/7288 e 0 to 0 dl 1787827218 ref 17 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.875399:0:10957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e7c from it ffff8ab3d0bd7190 00010000:00010000:3.0:1787827202.875401:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031800/0x268e1a602cc0e7c lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d612b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.875406:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031800/0x268e1a602cc0e7c lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d612b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827202.875414:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0532940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280339456:192.168.203.124@tcp:63:ll_sa.0 00010000:00010000:3.0:1787827202.875433:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7600/0x268e1a602cc0e0c lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5cb6 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.875446:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x7b:0x0](ffff8ab3d1f8c2c8) 00000080:00200000:3.0:1787827202.875450:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x7b:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.875464:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x7b:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.875467:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x7b:0x0](ffff8ab3d1f8c2c8) for lock 0x268e1a602cc0e0c 00000080:00010000:3.0:1787827202.875478:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e0c from it ffff8ab3d0980990 00010000:00010000:3.0:1787827202.875480:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7600/0x268e1a602cc0e0c lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5cb6 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.875486:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7600/0x268e1a602cc0e0c lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5cb6 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.875501:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc0e13 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d18 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.875514:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x8a:0x0](ffff8ab3d1f8aa08) 00000080:00200000:3.0:1787827202.875521:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x8a:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.875533:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x8a:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.875538:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x8a:0x0](ffff8ab3d1f8aa08) for lock 0x268e1a602cc0e13 00000080:00010000:3.0:1787827202.875543:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e13 from it ffff8ab3d0980d90 00010000:00010000:3.0:1787827202.875546:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc0e13 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5d18 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.875559:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc0e13 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5d18 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.875584:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7800/0x268e1a602cc0e1a lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d42 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.875595:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x33:0x0](ffff8ab3d1f880c8) 00000080:00200000:3.0:1787827202.875624:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x33:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.875627:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x33:0x0](ffff8ab3d1f880c8) for lock 0x268e1a602cc0e1a 00000080:00010000:3.0:1787827202.875636:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e1a from it ffff8ab3d0980190 00010000:00010000:3.0:1787827202.875638:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7800/0x268e1a602cc0e1a lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5d42 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.875644:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7800/0x268e1a602cc0e1a lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5d42 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.875663:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc0e21 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d5e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.875681:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0xa8:0x0](ffff8ab3d1f89148) 00000080:00200000:3.0:1787827202.875687:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0xa8:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.875702:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0xa8:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.875708:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0xa8:0x0](ffff8ab3d1f89148) for lock 0x268e1a602cc0e21 00000080:00010000:3.0:1787827202.875714:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e21 from it ffff8ab3d0981d90 00010000:00010000:3.0:1787827202.875718:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc0e21 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5d5e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.875727:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc0e21 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5d5e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.875746:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6a00/0x268e1a602cc0e28 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5d96 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.875758:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x6e:0x0](ffff8ab3d1f8d348) 00000080:00200000:3.0:1787827202.875761:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x6e:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.875775:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x6e:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.875781:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x6e:0x0](ffff8ab3d1f8d348) for lock 0x268e1a602cc0e28 00000080:00010000:3.0:1787827202.875786:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e28 from it ffff8ab3d0983190 00010000:00010000:3.0:1787827202.875789:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6a00/0x268e1a602cc0e28 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5d96 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.875798:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6a00/0x268e1a602cc0e28 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x6e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5d96 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.875829:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc0e2f lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5dce expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.875848:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x4d:0x0](ffff8ab3d1f88908) 00000080:00200000:3.0:1787827202.875876:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x4d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.875881:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x4d:0x0](ffff8ab3d1f88908) for lock 0x268e1a602cc0e2f 00000080:00010000:3.0:1787827202.875916:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e2f from it ffff8ab3d0982590 00010000:00010000:3.0:1787827202.875920:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc0e2f lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5dce expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.875932:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc0e2f lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5dce expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.875959:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc0e36 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e22 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.875974:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0xa6:0x0](ffff8ab3d1f8ec08) 00000080:00200000:3.0:1787827202.875981:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0xa6:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.876034:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0xa6:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.876039:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0xa6:0x0](ffff8ab3d1f8ec08) for lock 0x268e1a602cc0e36 00000080:00010000:3.0:1787827202.876051:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e36 from it ffff8ab3d0981990 00010000:00010000:3.0:1787827202.876054:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc0e36 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5e22 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.876060:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc0e36 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5e22 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.876083:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f400/0x268e1a602cc0e3d lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e61 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.876119:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0xa0:0x0](ffff8ab3d1f8db88) 00000080:00200000:3.0:1787827202.876125:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0xa0:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.876137:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0xa0:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.876141:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0xa0:0x0](ffff8ab3d1f8db88) for lock 0x268e1a602cc0e3d 00000080:00010000:3.0:1787827202.876150:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e3d from it ffff8ab3d0bd5990 00010000:00010000:3.0:1787827202.876154:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f400/0x268e1a602cc0e3d lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5e61 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.876160:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f400/0x268e1a602cc0e3d lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5e61 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.876266:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fc00/0x268e1a602cc0e44 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ec3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.876334:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x9c:0x0](ffff8ab3d1f8f448) 00000080:00200000:3.0:1787827202.876338:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x9c:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.876348:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x9c:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.876351:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x9c:0x0](ffff8ab3d1f8f448) for lock 0x268e1a602cc0e44 00000080:00010000:3.0:1787827202.876361:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e44 from it ffff8ab3d0bd6990 00010000:00010000:3.0:1787827202.876364:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fc00/0x268e1a602cc0e44 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5ec3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.876370:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fc00/0x268e1a602cc0e44 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5ec3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.876388:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc0e4b lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f1e expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.876409:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x43:0x0](ffff8ab3d1f8cb08) 00000080:00200000:3.0:1787827202.876438:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x43:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.876442:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x43:0x0](ffff8ab3d1f8cb08) for lock 0x268e1a602cc0e4b 00000080:00010000:3.0:1787827202.876451:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e4b from it ffff8ab3d0bd7990 00010000:00010000:3.0:1787827202.876455:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc0e4b lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5f1e expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.876461:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc0e4b lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5f1e expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.876484:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e800/0x268e1a602cc0e52 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f5d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.876574:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x75:0x0](ffff8ab3f12e8908) 00000080:00200000:3.0:1787827202.876583:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x75:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.876605:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x75:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.876615:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x75:0x0](ffff8ab3f12e8908) for lock 0x268e1a602cc0e52 00000080:00010000:3.0:1787827202.876696:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e52 from it ffff8ab3d0bd6190 00010000:00010000:3.0:1787827202.876703:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e800/0x268e1a602cc0e52 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5f5d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.876714:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e800/0x268e1a602cc0e52 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5f5d expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.876824:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e600/0x268e1a602cc0e59 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f9c expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.876862:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x29:0x0](ffff8ab3f12ea1c8) 00000080:00200000:3.0:1787827202.876939:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x29:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.876946:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x29:0x0](ffff8ab3f12ea1c8) for lock 0x268e1a602cc0e59 00000080:00010000:3.0:1787827202.877059:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e59 from it ffff8ab3d0bd5190 00010000:00010000:3.0:1787827202.877066:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e600/0x268e1a602cc0e59 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5f9c expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.877082:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e600/0x268e1a602cc0e59 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5f9c expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.877104:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc0e60 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5ff7 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.877152:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x19:0x0](ffff8ab3f12ecb08) 00000080:00200000:3.0:1787827202.877197:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x19:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.877270:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x19:0x0](ffff8ab3f12ecb08) for lock 0x268e1a602cc0e60 00000080:00010000:3.0:1787827202.877324:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e60 from it ffff8ab3d0bd4d90 00010000:00010000:3.0:1787827202.877327:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc0e60 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5ff7 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.877334:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc0e60 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5ff7 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.877360:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f600/0x268e1a602cc0e67 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6044 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.877426:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x34:0x0](ffff8ab3f12eba88) 00000080:00200000:3.0:1787827202.877459:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x34:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.877467:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x34:0x0](ffff8ab3f12eba88) for lock 0x268e1a602cc0e67 00000080:00010000:3.0:1787827202.877472:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e67 from it ffff8ab3d0bd4990 00010000:00010000:3.0:1787827202.877475:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f600/0x268e1a602cc0e67 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6044 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.877491:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166f600/0x268e1a602cc0e67 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d6044 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.877525:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc0e6e lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d609f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.877553:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x74:0x0](ffff8ab3f12ed348) 00000080:00200000:3.0:1787827202.877561:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x74:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.877580:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x74:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.877590:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x74:0x0](ffff8ab3f12ed348) for lock 0x268e1a602cc0e6e 00000080:00010000:3.0:1787827202.877605:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e6e from it ffff8ab3d0bd7d90 00010000:00010000:3.0:1787827202.877615:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc0e6e lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d609f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.877643:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc0e6e lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d609f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.877684:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc0e75 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60ec expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.877696:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0xd:0x0](ffff8ab3f12eec08) 00000080:00200000:3.0:1787827202.877770:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0xd:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.877776:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0xd:0x0](ffff8ab3f12eec08) for lock 0x268e1a602cc0e75 00000080:00010000:3.0:1787827202.877785:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e75 from it ffff8ab3d0bd5590 00010000:00010000:3.0:1787827202.877789:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc0e75 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d60ec expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.877795:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc0e75 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d60ec expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.877814:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031800/0x268e1a602cc0e7c lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d612b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.877828:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x57:0x0](ffff8ab3f12e9988) 00000080:00200000:3.0:1787827202.877832:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x57:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.877846:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x57:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.877849:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x57:0x0](ffff8ab3f12e9988) for lock 0x268e1a602cc0e7c 00000080:00010000:3.0:1787827202.877853:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e7c from it ffff8ab3d0bd7190 00010000:00010000:3.0:1787827202.877856:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031800/0x268e1a602cc0e7c lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d612b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.877969:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031800/0x268e1a602cc0e7c lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d612b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.879097:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b1fbdf37/0x268e1a602cc0f94 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879178:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b1fbdf37/0x268e1a602cc0f94 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d631c expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879193:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b1fbdf37/0x268e1a602cc0f94 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d631c expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879214:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b1fbdf37/0x268e1a602cc0f94 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d631c expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827202.879289:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 4207, setting rss=4207; kms=4207, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b1fbdf37/0x268e1a602cc0f94 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d631c expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879320:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b1fbdf37/0x268e1a602cc0f94 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d631c expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879328:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b1fbdf37/0x268e1a602cc0f94 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d631c expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879336:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b1fbdf37/0x268e1a602cc0f94 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d631c expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879356:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b1fbdf37/0x268e1a602cc0f94 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d631c expref: -99 pid: 48411 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827202.879375:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0532d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280342912:192.168.203.124@tcp:101:ll_agl.0 00010000:00010000:3.0:1787827202.879416:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031400/0x268e1a602cc0e8a lrc: 4/1,0 mode: --/PR res: [0x240000402:0x8e:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5db9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879444:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031400/0x268e1a602cc0e8a lrc: 4/1,0 mode: --/PR res: [0x240000402:0x8e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5db9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879456:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031400/0x268e1a602cc0e8a lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x8e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5db9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879471:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031400/0x268e1a602cc0e8a lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x8e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5db9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879488:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031400/0x268e1a602cc0e8a lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x8e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5db9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879497:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031400/0x268e1a602cc0e8a lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x8e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5db9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879507:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031400/0x268e1a602cc0e8a lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x8e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5db9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.879549:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 1 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.879564:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e8a from it ffff8ab3d0bd4590 00010000:00010000:3.0:1787827202.879568:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031400/0x268e1a602cc0e8a lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x8e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5db9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879584:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031400/0x268e1a602cc0e8a lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x8e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5db9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.879625:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc0e91 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 31 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5df8 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879642:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc0e91 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x21:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5df8 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879650:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc0e91 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5df8 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879657:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc0e91 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5df8 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879667:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc0e91 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5df8 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879678:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc0e91 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5df8 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879692:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc0e91 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5df8 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879700:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc0e91 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5df8 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.879710:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 2 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.879760:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc0e91 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5df8 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.879768:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e91 from it ffff8ab3d0bd6590 00010000:00010000:3.0:1787827202.879770:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc0e91 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5df8 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879776:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc0e91 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5df8 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.879787:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030a00/0x268e1a602cc0e98 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 30 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e3e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879794:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030a00/0x268e1a602cc0e98 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x17:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e3e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879798:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030a00/0x268e1a602cc0e98 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e3e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879808:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030a00/0x268e1a602cc0e98 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e3e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879813:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030a00/0x268e1a602cc0e98 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e3e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879818:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030a00/0x268e1a602cc0e98 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e3e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879822:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030a00/0x268e1a602cc0e98 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e3e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879827:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030a00/0x268e1a602cc0e98 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e3e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.879833:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 3 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.879838:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030a00/0x268e1a602cc0e98 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e3e expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.879843:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e98 from it ffff8ab3d0bd6d90 00010000:00010000:3.0:1787827202.879845:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030a00/0x268e1a602cc0e98 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e3e expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879851:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030a00/0x268e1a602cc0e98 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e3e expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.879861:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc0e9f lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 29 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e84 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879868:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc0e9f lrc: 4/1,0 mode: --/PR res: [0x240000402:0x88:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e84 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879873:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc0e9f lrc: 4/1,0 mode: --/PR res: [0x240000402:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e84 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879877:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc0e9f lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e84 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879881:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc0e9f lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e84 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879956:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc0e9f lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e84 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879965:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc0e9f lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e84 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.879971:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc0e9f lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e84 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.879981:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 4 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.879997:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e9f from it ffff8ab3d0bd4190 00010000:00010000:3.0:1787827202.880000:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc0e9f lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e84 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880008:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc0e9f lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5e84 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.880025:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc0ea6 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 28 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5eb5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880037:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc0ea6 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xa9:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5eb5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880044:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc0ea6 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5eb5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880053:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc0ea6 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5eb5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880061:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc0ea6 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5eb5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880070:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc0ea6 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5eb5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880078:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc0ea6 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5eb5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880091:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc0ea6 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5eb5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.880101:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 5 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.880125:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ea6 from it ffff8ab3f0495190 00010000:00010000:3.0:1787827202.880129:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc0ea6 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5eb5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880137:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc0ea6 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5eb5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.880152:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc0ead lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 27 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5eed expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880165:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc0ead lrc: 4/1,0 mode: --/PR res: [0x240000402:0x92:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5eed expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880173:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc0ead lrc: 4/1,0 mode: --/PR res: [0x240000402:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5eed expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880181:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc0ead lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5eed expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880190:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc0ead lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5eed expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880203:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc0ead lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5eed expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880211:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc0ead lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5eed expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880219:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc0ead lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5eed expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.880229:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 6 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.880238:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ead from it ffff8ab3f0494d90 00010000:00010000:3.0:1787827202.880241:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc0ead lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5eed expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880250:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc0ead lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5eed expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.880264:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031200/0x268e1a602cc0eb4 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f33 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880289:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031200/0x268e1a602cc0eb4 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x69:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f33 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880298:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031200/0x268e1a602cc0eb4 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x69:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f33 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880312:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031200/0x268e1a602cc0eb4 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x69:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f33 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880322:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031200/0x268e1a602cc0eb4 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x69:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f33 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880331:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031200/0x268e1a602cc0eb4 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x69:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f33 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880338:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031200/0x268e1a602cc0eb4 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x69:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f33 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880342:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031200/0x268e1a602cc0eb4 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x69:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f33 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.880348:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 7 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.880354:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0eb4 from it ffff8ab3f0494990 00010000:00010000:3.0:1787827202.880356:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031200/0x268e1a602cc0eb4 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x69:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f33 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880361:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031200/0x268e1a602cc0eb4 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x69:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f33 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.880459:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031a00/0x268e1a602cc0ebb lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f6b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880474:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031a00/0x268e1a602cc0ebb lrc: 4/1,0 mode: --/PR res: [0x240000402:0x5d:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f6b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880480:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031a00/0x268e1a602cc0ebb lrc: 4/1,0 mode: --/PR res: [0x240000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f6b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880486:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031a00/0x268e1a602cc0ebb lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f6b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880535:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031a00/0x268e1a602cc0ebb lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f6b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880557:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031a00/0x268e1a602cc0ebb lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f6b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880583:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031a00/0x268e1a602cc0ebb lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f6b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880599:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031a00/0x268e1a602cc0ebb lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f6b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.880621:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 8 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.880648:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ebb from it ffff8ab3f0497d90 00010000:00010000:3.0:1787827202.880655:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031a00/0x268e1a602cc0ebb lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f6b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880707:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031a00/0x268e1a602cc0ebb lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5f6b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.880745:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030c00/0x268e1a602cc0ec2 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5fa3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880763:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030c00/0x268e1a602cc0ec2 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x1d:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5fa3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880769:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030c00/0x268e1a602cc0ec2 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5fa3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880778:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030c00/0x268e1a602cc0ec2 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5fa3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880786:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030c00/0x268e1a602cc0ec2 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5fa3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880795:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030c00/0x268e1a602cc0ec2 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5fa3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880801:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030c00/0x268e1a602cc0ec2 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5fa3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880812:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030c00/0x268e1a602cc0ec2 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5fa3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.880844:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 9 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.880871:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030c00/0x268e1a602cc0ec2 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5fa3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.880908:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ec2 from it ffff8ab3f0495590 00010000:00010000:3.0:1787827202.880912:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030c00/0x268e1a602cc0ec2 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5fa3 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880918:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030c00/0x268e1a602cc0ec2 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5fa3 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.880932:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc0ec9 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5fdb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880944:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc0ec9 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x65:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5fdb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880950:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc0ec9 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5fdb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880955:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc0ec9 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5fdb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880961:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc0ec9 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5fdb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880970:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc0ec9 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5fdb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880975:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc0ec9 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5fdb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.880980:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc0ec9 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5fdb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.880987:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 10 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.880992:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ec9 from it ffff8ab3f0497190 00010000:00010000:3.0:1787827202.880994:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc0ec9 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5fdb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881000:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc0ec9 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d5fdb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.881012:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc0ed0 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6013 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881019:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc0ed0 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x46:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6013 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881024:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc0ed0 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6013 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881032:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc0ed0 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6013 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881037:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc0ed0 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6013 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881076:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc0ed0 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6013 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881082:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc0ed0 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6013 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881086:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc0ed0 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6013 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.881092:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 11 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.881098:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc0ed0 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6013 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.881104:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ed0 from it ffff8ab3f0495d90 00010000:00010000:3.0:1787827202.881106:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc0ed0 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6013 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881128:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc0ed0 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6013 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.881143:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc0ed7 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d604b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881154:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc0ed7 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x1b:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d604b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881163:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc0ed7 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d604b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881171:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc0ed7 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d604b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881180:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc0ed7 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d604b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881189:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc0ed7 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d604b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881198:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc0ed7 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d604b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881206:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc0ed7 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d604b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.881217:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 12 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.881234:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc0ed7 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d604b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.881243:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ed7 from it ffff8ab3f0494190 00010000:00010000:3.0:1787827202.881247:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc0ed7 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d604b expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881252:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc0ed7 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d604b expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.881264:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc0ede lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6075 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881270:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc0ede lrc: 4/1,0 mode: --/PR res: [0x240000402:0x72:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6075 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881276:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc0ede lrc: 4/1,0 mode: --/PR res: [0x240000402:0x72:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6075 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881284:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc0ede lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x72:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6075 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881293:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc0ede lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x72:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6075 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881302:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc0ede lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x72:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6075 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881317:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc0ede lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x72:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6075 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881325:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc0ede lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x72:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6075 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.881334:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 13 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.881340:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ede from it ffff8ab3f0496d90 00010000:00010000:3.0:1787827202.881342:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc0ede lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x72:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6075 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881347:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc0ede lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x72:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6075 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.881357:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc0ee5 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60bb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881365:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc0ee5 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x42:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60bb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881370:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc0ee5 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60bb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881375:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc0ee5 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60bb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881383:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc0ee5 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60bb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881388:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc0ee5 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60bb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881392:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc0ee5 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60bb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881396:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc0ee5 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60bb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.881401:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 14 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.881406:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc0ee5 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60bb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.881411:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ee5 from it ffff8ab3f0497590 00010000:00010000:3.0:1787827202.881413:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc0ee5 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60bb expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881417:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc0ee5 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60bb expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.881425:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc0eec lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60f3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881435:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc0eec lrc: 4/1,0 mode: --/PR res: [0x240000402:0x64:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60f3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881440:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc0eec lrc: 4/1,0 mode: --/PR res: [0x240000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60f3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881444:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc0eec lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60f3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881448:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc0eec lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60f3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881453:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc0eec lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60f3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881458:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc0eec lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60f3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881462:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc0eec lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60f3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.881467:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 15 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.881472:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0eec from it ffff8ab3f0494590 00010000:00010000:3.0:1787827202.881474:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc0eec lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60f3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881482:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc0eec lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d60f3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.881490:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1400/0x268e1a602cc0ef3 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6124 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881496:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1400/0x268e1a602cc0ef3 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x15:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6124 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881503:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1400/0x268e1a602cc0ef3 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6124 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881512:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1400/0x268e1a602cc0ef3 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6124 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881520:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1400/0x268e1a602cc0ef3 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6124 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881529:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1400/0x268e1a602cc0ef3 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6124 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881538:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1400/0x268e1a602cc0ef3 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6124 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881545:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1400/0x268e1a602cc0ef3 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6124 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.881560:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 16 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.881570:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1400/0x268e1a602cc0ef3 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6124 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.881580:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ef3 from it ffff8ab3f0496590 00010000:00010000:3.0:1787827202.881583:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1400/0x268e1a602cc0ef3 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6124 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881588:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1400/0x268e1a602cc0ef3 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6124 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.881598:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0c00/0x268e1a602cc0efa lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d615c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881606:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0c00/0x268e1a602cc0efa lrc: 4/1,0 mode: --/PR res: [0x240000402:0x85:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d615c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881611:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0c00/0x268e1a602cc0efa lrc: 4/1,0 mode: --/PR res: [0x240000402:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d615c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881616:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0c00/0x268e1a602cc0efa lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d615c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881621:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0c00/0x268e1a602cc0efa lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d615c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881629:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0c00/0x268e1a602cc0efa lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d615c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881634:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0c00/0x268e1a602cc0efa lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d615c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881638:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0c00/0x268e1a602cc0efa lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d615c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.881643:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 17 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.881648:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0efa from it ffff8ab3f0495990 00010000:00010000:3.0:1787827202.881650:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0c00/0x268e1a602cc0efa lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d615c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881655:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0c00/0x268e1a602cc0efa lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d615c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.881668:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc0f01 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d617f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881679:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc0f01 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xa1:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d617f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881686:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc0f01 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d617f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881699:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc0f01 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d617f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881708:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc0f01 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d617f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881715:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc0f01 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d617f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881720:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc0f01 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d617f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881724:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc0f01 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d617f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.881730:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 18 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.881735:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f01 from it ffff8ab3f0496990 00010000:00010000:3.0:1787827202.881737:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc0f01 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d617f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881742:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc0f01 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d617f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.881751:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0200/0x268e1a602cc0f08 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d619b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881762:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0200/0x268e1a602cc0f08 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xa4:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d619b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881767:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0200/0x268e1a602cc0f08 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d619b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881772:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0200/0x268e1a602cc0f08 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d619b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881776:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0200/0x268e1a602cc0f08 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d619b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881781:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0200/0x268e1a602cc0f08 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d619b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881785:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0200/0x268e1a602cc0f08 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d619b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881793:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0200/0x268e1a602cc0f08 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d619b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.881801:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 19 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.881809:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f08 from it ffff8ab3f0497990 00010000:00010000:3.0:1787827202.881812:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0200/0x268e1a602cc0f08 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d619b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881825:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0200/0x268e1a602cc0f08 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d619b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.881840:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc0f0f lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61b7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881849:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc0f0f lrc: 4/1,0 mode: --/PR res: [0x240000402:0x4c:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61b7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881854:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc0f0f lrc: 4/1,0 mode: --/PR res: [0x240000402:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61b7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881859:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc0f0f lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61b7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881864:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc0f0f lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61b7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881869:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc0f0f lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61b7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881873:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc0f0f lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61b7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881878:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc0f0f lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61b7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.881910:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 20 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.881917:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc0f0f lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61b7 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.881924:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f0f from it ffff8ab3f0496190 00010000:00010000:3.0:1787827202.881926:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc0f0f lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61b7 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881931:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc0f0f lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61b7 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.881941:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0600/0x268e1a602cc0f16 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61d3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881948:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0600/0x268e1a602cc0f16 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x56:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61d3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881952:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0600/0x268e1a602cc0f16 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61d3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881958:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0600/0x268e1a602cc0f16 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61d3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881962:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0600/0x268e1a602cc0f16 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61d3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881971:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0600/0x268e1a602cc0f16 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61d3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881975:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0600/0x268e1a602cc0f16 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61d3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881979:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0600/0x268e1a602cc0f16 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61d3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.881984:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 21 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.881989:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f16 from it ffff8ab3f04e0190 00010000:00010000:3.0:1787827202.881991:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0600/0x268e1a602cc0f16 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61d3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.881995:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0600/0x268e1a602cc0f16 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61d3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.882002:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1000/0x268e1a602cc0f1d lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61ef expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882008:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1000/0x268e1a602cc0f1d lrc: 4/1,0 mode: --/PR res: [0x240000402:0x12:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61ef expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882012:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1000/0x268e1a602cc0f1d lrc: 4/1,0 mode: --/PR res: [0x240000402:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61ef expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882020:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1000/0x268e1a602cc0f1d lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61ef expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882024:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1000/0x268e1a602cc0f1d lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61ef expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882029:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1000/0x268e1a602cc0f1d lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61ef expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882033:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1000/0x268e1a602cc0f1d lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61ef expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882037:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1000/0x268e1a602cc0f1d lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61ef expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.882042:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 22 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.882047:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1000/0x268e1a602cc0f1d lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61ef expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.882052:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f1d from it ffff8ab3f04e2d90 00010000:00010000:3.0:1787827202.882053:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1000/0x268e1a602cc0f1d lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61ef expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882058:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1000/0x268e1a602cc0f1d lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d61ef expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.882068:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1200/0x268e1a602cc0f24 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6220 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882083:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1200/0x268e1a602cc0f24 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x53:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6220 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882088:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1200/0x268e1a602cc0f24 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6220 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882093:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1200/0x268e1a602cc0f24 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6220 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882098:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1200/0x268e1a602cc0f24 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6220 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882104:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1200/0x268e1a602cc0f24 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6220 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882124:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1200/0x268e1a602cc0f24 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6220 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882186:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1200/0x268e1a602cc0f24 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6220 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.882194:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 23 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.882205:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f24 from it ffff8ab3f04e3590 00010000:00010000:3.0:1787827202.882208:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1200/0x268e1a602cc0f24 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6220 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882214:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1200/0x268e1a602cc0f24 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6220 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.882228:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc0f2b lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6243 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882238:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc0f2b lrc: 4/1,0 mode: --/PR res: [0x240000402:0x83:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6243 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882243:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc0f2b lrc: 4/1,0 mode: --/PR res: [0x240000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6243 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882249:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc0f2b lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6243 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882255:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc0f2b lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6243 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882260:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc0f2b lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6243 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882275:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc0f2b lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6243 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882284:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc0f2b lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6243 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.882293:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 24 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.882303:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f2b from it ffff8ab3f04e0590 00010000:00010000:3.0:1787827202.882305:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc0f2b lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6243 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882314:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc0f2b lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6243 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.882332:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc0f32 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6266 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882341:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc0f32 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x79:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6266 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882346:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc0f32 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6266 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882352:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc0f32 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6266 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882357:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc0f32 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6266 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882369:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc0f32 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6266 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882381:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc0f32 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6266 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882390:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc0f32 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6266 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.882403:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 25 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.882427:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f32 from it ffff8ab3f04e2590 00010000:00010000:3.0:1787827202.882432:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc0f32 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6266 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882441:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc0f32 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6266 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.882459:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0e00/0x268e1a602cc0f39 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6289 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882469:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0e00/0x268e1a602cc0f39 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xb1:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6289 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882476:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0e00/0x268e1a602cc0f39 lrc: 4/1,0 mode: --/PR res: [0x240000402:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6289 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882485:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0e00/0x268e1a602cc0f39 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6289 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882494:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0e00/0x268e1a602cc0f39 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6289 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882504:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0e00/0x268e1a602cc0f39 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6289 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882515:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0e00/0x268e1a602cc0f39 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6289 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882531:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0e00/0x268e1a602cc0f39 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6289 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.882555:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 26 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.882563:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f39 from it ffff8ab3f04e1990 00010000:00010000:3.0:1787827202.882571:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0e00/0x268e1a602cc0f39 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6289 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882591:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0e00/0x268e1a602cc0f39 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6289 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.882617:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc0f40 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d62ac expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882636:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc0f40 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x4a:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d62ac expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882646:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc0f40 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d62ac expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882654:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc0f40 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d62ac expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882663:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc0f40 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d62ac expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882672:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc0f40 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d62ac expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882680:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc0f40 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d62ac expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882687:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc0f40 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d62ac expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.882697:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 27 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.882706:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc0f40 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d62ac expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.882721:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f40 from it ffff8ab3f04e2990 00010000:00010000:3.0:1787827202.882725:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc0f40 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d62ac expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882733:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc0f40 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d62ac expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.882749:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc0f47 lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d62e4 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882760:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc0f47 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x96:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d62e4 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882811:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc0f47 lrc: 4/1,0 mode: --/PR res: [0x240000402:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d62e4 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882819:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc0f47 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d62e4 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882828:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc0f47 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d62e4 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882838:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc0f47 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d62e4 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882844:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc0f47 lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d62e4 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882852:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc0f47 lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d62e4 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.882859:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 28 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:3.0:1787827202.882864:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f47 from it ffff8ab3f04e3990 00010000:00010000:3.0:1787827202.882866:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc0f47 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d62e4 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882871:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc0f47 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d62e4 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.882881:0:10954:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc0f4e lrc: 4/1,0 mode: --/CR res: [0x240000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6315 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882912:0:10954:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc0f4e lrc: 4/1,0 mode: --/PR res: [0x240000402:0x4:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6315 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882917:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc0f4e lrc: 4/1,0 mode: --/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6315 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882923:0:10954:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc0f4e lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6315 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882928:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc0f4e lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6315 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882938:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc0f4e lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6315 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882943:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc0f4e lrc: 5/2,0 mode: PR/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6315 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882947:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc0f4e lrc: 4/1,0 mode: PR/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6315 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827202.882953:0:10954:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8ab3f0531e00 x1874671280339840/t0(0) o63->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:24/4 lens 280/12664 e 0 to 0 dl 1787827218 ref 29 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1787827202.882958:0:10954:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc0f4e lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6315 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1787827202.882963:0:10954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f4e from it ffff8ab3f04e2190 00010000:00010000:3.0:1787827202.882965:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc0f4e lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6315 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827202.882970:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc0f4e lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6315 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827202.882982:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0531e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280339840:192.168.203.124@tcp:63:ll_sa.0 00010000:00010000:3.0:1787827202.883005:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031400/0x268e1a602cc0e8a lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x8e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5db9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.883023:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x8e:0x0](ffff8ab3f12e80c8) 00000080:00200000:3.0:1787827202.883028:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x8e:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.883045:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x8e:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.883050:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x8e:0x0](ffff8ab3f12e80c8) for lock 0x268e1a602cc0e8a 00000080:00010000:3.0:1787827202.883060:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e8a from it ffff8ab3d0bd4590 00010000:00010000:3.0:1787827202.883063:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031400/0x268e1a602cc0e8a lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x8e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5db9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.883069:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031400/0x268e1a602cc0e8a lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x8e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5db9 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.883084:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc0e91 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5df8 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.883095:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x21:0x0](ffff8ab3f12e9148) 00000080:00200000:3.0:1787827202.883140:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x21:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.883145:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x21:0x0](ffff8ab3f12e9148) for lock 0x268e1a602cc0e91 00000080:00010000:3.0:1787827202.883153:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e91 from it ffff8ab3d0bd6590 00010000:00010000:3.0:1787827202.883155:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc0e91 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5df8 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.883162:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc0e91 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5df8 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.883177:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030a00/0x268e1a602cc0e98 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e3e expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.883190:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x17:0x0](ffff8ab3f12ec2c8) 00000080:00200000:3.0:1787827202.883214:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x17:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.883218:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x17:0x0](ffff8ab3f12ec2c8) for lock 0x268e1a602cc0e98 00000080:00010000:3.0:1787827202.883222:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e98 from it ffff8ab3d0bd6d90 00010000:00010000:3.0:1787827202.883226:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030a00/0x268e1a602cc0e98 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5e3e expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.883247:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030a00/0x268e1a602cc0e98 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5e3e expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.883261:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc0e9f lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5e84 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.883272:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x88:0x0](ffff8ab3f12edb88) 00000080:00200000:3.0:1787827202.883275:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x88:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.883282:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x88:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.883285:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x88:0x0](ffff8ab3f12edb88) for lock 0x268e1a602cc0e9f 00000080:00010000:3.0:1787827202.883288:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0e9f from it ffff8ab3d0bd4190 00010000:00010000:3.0:1787827202.883294:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc0e9f lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5e84 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.883310:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc0e9f lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5e84 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.883332:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc0ea6 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5eb5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.883345:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0xa9:0x0](ffff8ab3f12ee3c8) 00000080:00200000:3.0:1787827202.883349:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0xa9:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.883363:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0xa9:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.883368:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0xa9:0x0](ffff8ab3f12ee3c8) for lock 0x268e1a602cc0ea6 00000080:00010000:3.0:1787827202.883371:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ea6 from it ffff8ab3f0495190 00010000:00010000:3.0:1787827202.883375:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc0ea6 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5eb5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.883384:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc0ea6 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5eb5 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.883400:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc0ead lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5eed expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.883412:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x92:0x0](ffff8ab3f12eaa08) 00000080:00200000:3.0:1787827202.883417:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x92:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.883430:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x92:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.883437:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x92:0x0](ffff8ab3f12eaa08) for lock 0x268e1a602cc0ead 00000080:00010000:3.0:1787827202.883444:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ead from it ffff8ab3f0494d90 00010000:00010000:3.0:1787827202.883448:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc0ead lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5eed expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.883454:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc0ead lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5eed expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.883467:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031200/0x268e1a602cc0eb4 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x69:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f33 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.883610:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x69:0x0](ffff8ab3f12ef448) 00000080:00200000:3.0:1787827202.883615:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x69:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.883628:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x69:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.883633:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x69:0x0](ffff8ab3f12ef448) for lock 0x268e1a602cc0eb4 00000080:00010000:3.0:1787827202.883689:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0eb4 from it ffff8ab3f0494990 00010000:00010000:3.0:1787827202.883745:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031200/0x268e1a602cc0eb4 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x69:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5f33 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.883752:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031200/0x268e1a602cc0eb4 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x69:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5f33 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.883766:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031a00/0x268e1a602cc0ebb lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5f6b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.883776:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x5d:0x0](ffff8ab3f12eb248) 00000080:00200000:3.0:1787827202.883779:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x5d:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.883790:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x5d:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.883794:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x5d:0x0](ffff8ab3f12eb248) for lock 0x268e1a602cc0ebb 00000080:00010000:3.0:1787827202.883807:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ebb from it ffff8ab3f0497d90 00010000:00010000:3.0:1787827202.883811:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031a00/0x268e1a602cc0ebb lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5f6b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.883820:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031a00/0x268e1a602cc0ebb lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5f6b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.883844:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030c00/0x268e1a602cc0ec2 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5fa3 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.883860:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x1d:0x0](ffff8ab3f11eec08) 00000080:00200000:3.0:1787827202.883953:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x1d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.883957:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x1d:0x0](ffff8ab3f11eec08) for lock 0x268e1a602cc0ec2 00000080:00010000:3.0:1787827202.883969:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ec2 from it ffff8ab3f0495590 00010000:00010000:3.0:1787827202.883972:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030c00/0x268e1a602cc0ec2 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5fa3 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.883979:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030c00/0x268e1a602cc0ec2 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5fa3 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.883997:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc0ec9 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d5fdb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.884012:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x65:0x0](ffff8ab3f11ed348) 00000080:00200000:3.0:1787827202.884017:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x65:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.884030:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x65:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.884036:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x65:0x0](ffff8ab3f11ed348) for lock 0x268e1a602cc0ec9 00000080:00010000:3.0:1787827202.884041:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ec9 from it ffff8ab3f0497190 00010000:00010000:3.0:1787827202.884046:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc0ec9 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d5fdb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.884053:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc0ec9 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d5fdb expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.884070:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc0ed0 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6013 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.884099:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x46:0x0](ffff8ab3f11eb248) 00000080:00200000:3.0:1787827202.884155:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x46:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.884163:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x46:0x0](ffff8ab3f11eb248) for lock 0x268e1a602cc0ed0 00000080:00010000:3.0:1787827202.884174:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ed0 from it ffff8ab3f0495d90 00010000:00010000:3.0:1787827202.884184:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc0ed0 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6013 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.884215:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc0ed0 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d6013 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.884302:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc0ed7 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d604b expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.884439:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x1b:0x0](ffff8ab3f11e9988) 00000080:00200000:3.0:1787827202.884476:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x1b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.884483:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x1b:0x0](ffff8ab3f11e9988) for lock 0x268e1a602cc0ed7 00000080:00010000:3.0:1787827202.884590:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ed7 from it ffff8ab3f0494190 00010000:00010000:3.0:1787827202.884599:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc0ed7 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d604b expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.884624:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc0ed7 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d604b expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.884698:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc0ede lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x72:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6075 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.884769:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x72:0x0](ffff8ab3f11e8908) 00000080:00200000:3.0:1787827202.884773:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x72:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.884787:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x72:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.884791:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x72:0x0](ffff8ab3f11e8908) for lock 0x268e1a602cc0ede 00000080:00010000:3.0:1787827202.884810:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ede from it ffff8ab3f0496d90 00010000:00010000:3.0:1787827202.884815:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc0ede lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x72:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6075 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.884826:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc0ede lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x72:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d6075 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.884851:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc0ee5 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60bb expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.884946:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x42:0x0](ffff8ab3f11eaa08) 00000080:00200000:3.0:1787827202.885034:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x42:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.885038:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x42:0x0](ffff8ab3f11eaa08) for lock 0x268e1a602cc0ee5 00000080:00010000:3.0:1787827202.885048:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ee5 from it ffff8ab3f0497590 00010000:00010000:3.0:1787827202.885051:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc0ee5 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d60bb expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.885059:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc0ee5 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d60bb expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.885078:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc0eec lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d60f3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.885094:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x64:0x0](ffff8ab3f11ee3c8) 00000080:00200000:3.0:1787827202.885099:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x64:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.885177:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x64:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.885189:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x64:0x0](ffff8ab3f11ee3c8) for lock 0x268e1a602cc0eec 00000080:00010000:3.0:1787827202.885198:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0eec from it ffff8ab3f0494590 00010000:00010000:3.0:1787827202.885207:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc0eec lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d60f3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.885250:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc0eec lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d60f3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.885293:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1400/0x268e1a602cc0ef3 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6124 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.885314:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x15:0x0](ffff8ab3f11ecb08) 00000080:00200000:3.0:1787827202.885339:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x15:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.885341:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x15:0x0](ffff8ab3f11ecb08) for lock 0x268e1a602cc0ef3 00000080:00010000:3.0:1787827202.885398:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0ef3 from it ffff8ab3f0496590 00010000:00010000:3.0:1787827202.885403:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1400/0x268e1a602cc0ef3 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6124 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.885414:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1400/0x268e1a602cc0ef3 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d6124 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.885436:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0c00/0x268e1a602cc0efa lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d615c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.885448:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x85:0x0](ffff8ab3f11e80c8) 00000080:00200000:3.0:1787827202.885454:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x85:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.885467:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x85:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.885471:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x85:0x0](ffff8ab3f11e80c8) for lock 0x268e1a602cc0efa 00000080:00010000:3.0:1787827202.885475:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0efa from it ffff8ab3f0495990 00010000:00010000:3.0:1787827202.885478:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0c00/0x268e1a602cc0efa lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d615c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.885571:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0c00/0x268e1a602cc0efa lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d615c expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.885588:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc0f01 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d617f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.885602:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0xa1:0x0](ffff8ab3f11edb88) 00000080:00200000:3.0:1787827202.885605:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0xa1:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.885618:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0xa1:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.885622:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0xa1:0x0](ffff8ab3f11edb88) for lock 0x268e1a602cc0f01 00000080:00010000:3.0:1787827202.885633:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f01 from it ffff8ab3f0496990 00010000:00010000:3.0:1787827202.885636:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc0f01 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d617f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.885643:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc0f01 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d617f expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.885703:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0200/0x268e1a602cc0f08 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d619b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.885724:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0xa4:0x0](ffff8ab3f11e9148) 00000080:00200000:3.0:1787827202.885729:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0xa4:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.885742:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0xa4:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.885747:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0xa4:0x0](ffff8ab3f11e9148) for lock 0x268e1a602cc0f08 00000080:00010000:3.0:1787827202.885760:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f08 from it ffff8ab3f0497990 00010000:00010000:3.0:1787827202.885766:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0200/0x268e1a602cc0f08 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d619b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.885786:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0200/0x268e1a602cc0f08 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d619b expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.885831:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc0f0f lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61b7 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.885850:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x4c:0x0](ffff8ab3f11ea1c8) 00000080:00200000:3.0:1787827202.886010:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x4c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.886016:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x4c:0x0](ffff8ab3f11ea1c8) for lock 0x268e1a602cc0f0f 00000080:00010000:3.0:1787827202.886028:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f0f from it ffff8ab3f0496190 00010000:00010000:3.0:1787827202.886034:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc0f0f lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d61b7 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.886043:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc0f0f lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d61b7 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.886068:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0600/0x268e1a602cc0f16 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61d3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.886101:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x56:0x0](ffff8ab3f11ec2c8) 00000080:00200000:3.0:1787827202.886186:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x56:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.886205:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x56:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.886215:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x56:0x0](ffff8ab3f11ec2c8) for lock 0x268e1a602cc0f16 00000080:00010000:3.0:1787827202.886229:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f16 from it ffff8ab3f04e0190 00010000:00010000:3.0:1787827202.886233:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0600/0x268e1a602cc0f16 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d61d3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.886309:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0600/0x268e1a602cc0f16 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x56:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d61d3 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.886345:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1000/0x268e1a602cc0f1d lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d61ef expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.886378:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x12:0x0](ffff8ab3f11ef448) 00000080:00200000:3.0:1787827202.886413:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x12:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.886420:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x12:0x0](ffff8ab3f11ef448) for lock 0x268e1a602cc0f1d 00000080:00010000:3.0:1787827202.886425:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f1d from it ffff8ab3f04e2d90 00010000:00010000:3.0:1787827202.886428:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1000/0x268e1a602cc0f1d lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d61ef expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.886444:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1000/0x268e1a602cc0f1d lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d61ef expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.886464:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1200/0x268e1a602cc0f24 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6220 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.886480:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x53:0x0](ffff8ab3f1373a88) 00000080:00200000:3.0:1787827202.886484:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x53:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.886497:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x53:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.886501:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x53:0x0](ffff8ab3f1373a88) for lock 0x268e1a602cc0f24 00000080:00010000:3.0:1787827202.886504:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f24 from it ffff8ab3f04e3590 00010000:00010000:3.0:1787827202.886510:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1200/0x268e1a602cc0f24 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6220 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.886586:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1200/0x268e1a602cc0f24 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d6220 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.886606:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc0f2b lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6243 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.886625:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x83:0x0](ffff8ab3f1376c08) 00000080:00200000:3.0:1787827202.886629:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x83:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.886645:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x83:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.886650:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x83:0x0](ffff8ab3f1376c08) for lock 0x268e1a602cc0f2b 00000080:00010000:3.0:1787827202.886656:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f2b from it ffff8ab3f04e0590 00010000:00010000:3.0:1787827202.886660:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc0f2b lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6243 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.886673:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc0f2b lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d6243 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.886688:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc0f32 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6266 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.886751:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x79:0x0](ffff8ab3f13742c8) 00000080:00200000:3.0:1787827202.886758:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x79:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.886777:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x79:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.886784:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x79:0x0](ffff8ab3f13742c8) for lock 0x268e1a602cc0f32 00000080:00010000:3.0:1787827202.886797:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f32 from it ffff8ab3f04e2590 00010000:00010000:3.0:1787827202.886805:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc0f32 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6266 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.886829:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc0f32 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d6266 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.886850:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0e00/0x268e1a602cc0f39 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6289 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.886860:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0xb1:0x0](ffff8ab3f13721c8) 00000080:00200000:3.0:1787827202.886862:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0xb1:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.886991:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0xb1:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.886995:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0xb1:0x0](ffff8ab3f13721c8) for lock 0x268e1a602cc0f39 00000080:00010000:3.0:1787827202.887004:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f39 from it ffff8ab3f04e1990 00010000:00010000:3.0:1787827202.887007:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0e00/0x268e1a602cc0f39 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6289 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.887014:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0e00/0x268e1a602cc0f39 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d6289 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.887036:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc0f40 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d62ac expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.887055:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x4a:0x0](ffff8ab3f1373248) 00000080:00200000:3.0:1787827202.887081:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x4a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.887084:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x4a:0x0](ffff8ab3f1373248) for lock 0x268e1a602cc0f40 00000080:00010000:3.0:1787827202.887092:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f40 from it ffff8ab3f04e2990 00010000:00010000:3.0:1787827202.887095:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc0f40 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d62ac expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.887126:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc0f40 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d62ac expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.887148:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc0f47 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d62e4 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.887162:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x96:0x0](ffff8ab3f1370908) 00000080:00200000:3.0:1787827202.887168:0:14753:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240000402:0x96:0x0], updating i_size 0 00000080:00200000:3.0:1787827202.887181:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x96:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827202.887185:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x96:0x0](ffff8ab3f1370908) for lock 0x268e1a602cc0f47 00000080:00010000:3.0:1787827202.887194:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f47 from it ffff8ab3f04e3990 00010000:00010000:3.0:1787827202.887201:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc0f47 lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d62e4 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.887219:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc0f47 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d62e4 expref: -99 pid: 48410 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.887303:0:14753:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc0f4e lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6315 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827202.887322:0:14753:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240000402:0x4:0x0](ffff8ab3f1375b88) 00000080:00200000:3.0:1787827202.887365:0:14753:0:(file.c:6854:ll_layout_conf()) [0x240000402:0x4:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1787827202.887370:0:14753:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240000402:0x4:0x0](ffff8ab3f1375b88) for lock 0x268e1a602cc0f4e 00000080:00010000:3.0:1787827202.887464:0:14753:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc0f4e from it ffff8ab3f04e2190 00010000:00010000:3.0:1787827202.887469:0:14753:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc0f4e lrc: 3/1,0 mode: PR/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6315 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827202.887480:0:14753:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc0f4e lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d6315 expref: -99 pid: 48410 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827202.918016:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827202.958064:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0f04f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280358400:192.168.203.124@tcp:103:rm.0 00000100:00100000:3.0:1787827203.055076:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294762880 00000100:00100000:3.0:1787827203.055084:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294762880 00000100:00100000:3.0:1787827203.055136:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 1 00000100:00100000:3.0:1787827203.055146:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d0b62d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294762880:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.055156:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e15a8114/0x268e1a602cc1058 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x62:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d6b3b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.055199:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.055205:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d0b62d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294762880:12345-192.168.203.124@tcp:104: Request processed in 61us (174us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.055212:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 1 00000100:00100000:3.0:1787827203.062567:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0b63840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280364288:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.063306:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc1082 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.063318:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc1082 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6bb2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.063326:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc1082 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6bb2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.063334:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc1082 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6bb2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.063341:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc1082 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6bb2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.063348:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.063356:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc1082 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6bb2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.063362:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc1082 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6bb2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.063367:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc1082 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6bb2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.063373:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc1082 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d6bb2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.063383:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0b63840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280364288:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.064302:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294763136 00000100:00100000:3.0:1787827203.064307:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294763136 00000100:00100000:3.0:1787827203.064334:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 3 00000100:00100000:3.0:1787827203.064340:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3f04c6940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294763136:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.064347:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc1082 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x62:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d6bb2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.064401:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.064409:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3f04c6940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294763136:12345-192.168.203.124@tcp:104: Request processed in 68us (144us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.064420:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 3 00000100:00100000:3.0:1787827203.064660:0:10956:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1787827203.064676:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f062de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280364544:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.066139:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f062de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280364544:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.073943:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.073954:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294763392 00000100:00100000:3.0:1787827203.073959:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294763392 00000100:00100000:3.0:1787827203.073989:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 5 00000100:00100000:3.0:1787827203.073996:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c40d4050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294763392:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.074005:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c9541d35/0x268e1a602cc10a5 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x63:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d6c0d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.074047:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.074055:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c40d4050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294763392:12345-192.168.203.124@tcp:104: Request processed in 59us (169us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.074064:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 5 00000100:00100000:3.0:1787827203.084642:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0b62940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280366592:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.085449:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001055721d/0x268e1a602cc10c8 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.085465:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001055721d/0x268e1a602cc10c8 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6c68 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.085475:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001055721d/0x268e1a602cc10c8 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6c68 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.085482:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001055721d/0x268e1a602cc10c8 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6c68 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.085491:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001055721d/0x268e1a602cc10c8 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6c68 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.085498:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.085509:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001055721d/0x268e1a602cc10c8 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6c68 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.085515:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001055721d/0x268e1a602cc10c8 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6c68 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.085520:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001055721d/0x268e1a602cc10c8 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6c68 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.085525:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001055721d/0x268e1a602cc10c8 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d6c68 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.085540:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0b62940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280366592:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.086563:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294763648 00000100:00100000:3.0:1787827203.086569:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294763648 00000100:00100000:3.0:1787827203.086602:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 7 00000100:00100000:3.0:1787827203.086609:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3f04c6940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294763648:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.086618:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001055721d/0x268e1a602cc10c8 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x63:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d6c68 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.086656:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.086661:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3f04c6940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294763648:12345-192.168.203.124@tcp:104: Request processed in 53us (141us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.086668:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 7 00000100:00100000:3.0:1787827203.086817:0:10956:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1787827203.086831:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f062f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280366848:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.088367:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f062f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280366848:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.095425:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294763904 00000100:00100000:3.0:1787827203.095431:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294763904 00000100:00100000:3.0:1787827203.095487:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 9 00000100:00100000:3.0:1787827203.095499:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3f062ed00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294763904:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.095512:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000004f732e26/0x268e1a602cc10eb lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x64:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d6cc3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.095579:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.095585:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3f062ed00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294763904:12345-192.168.203.124@tcp:104: Request processed in 90us (200us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.095593:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 9 00010000:00010000:3.0:1787827203.103313:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc110e lrc: 6/2,0 mode: --/PR res: [0x280000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.103328:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc110e lrc: 6/2,0 mode: --/PR res: [0x280000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6d1e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.103340:0:10957:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc110e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6d1e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.103351:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc110e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6d1e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.103363:0:10957:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc110e lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6d1e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.103375:0:10957:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.103386:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc110e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6d1e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.103392:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc110e lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6d1e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.103398:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc110e lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6d1e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.103406:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc110e lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d6d1e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.103424:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f051cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280368896:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.104290:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.104300:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294764160 00000100:00100000:3.0:1787827203.104304:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294764160 00000100:00100000:3.0:1787827203.104338:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 11 00000100:00100000:3.0:1787827203.104348:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3765c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294764160:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.104359:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc110e lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x64:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d6d1e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.104406:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.104414:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3765c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294764160:12345-192.168.203.124@tcp:104: Request processed in 67us (166us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.104425:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 11 00000100:00100000:3.0:1787827203.106364:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f062e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280369152:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.114526:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294764416 00000100:00100000:3.0:1787827203.114533:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294764416 00000100:00100000:3.0:1787827203.114581:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 13 00000100:00100000:3.0:1787827203.114593:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3f04c6940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294764416:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.114605:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000003722e93c/0x268e1a602cc1131 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x65:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d6d79 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.114672:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.114681:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3f04c6940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294764416:12345-192.168.203.124@tcp:104: Request processed in 90us (217us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.114692:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 13 00000100:00100000:3.0:1787827203.122165:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280371200:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.122974:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000116384d8/0x268e1a602cc1154 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.122986:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000116384d8/0x268e1a602cc1154 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6dd4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.122994:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000116384d8/0x268e1a602cc1154 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6dd4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.123002:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000116384d8/0x268e1a602cc1154 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6dd4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.123010:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000116384d8/0x268e1a602cc1154 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6dd4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.123017:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.123026:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000116384d8/0x268e1a602cc1154 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6dd4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.123036:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000116384d8/0x268e1a602cc1154 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6dd4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.123042:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000116384d8/0x268e1a602cc1154 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6dd4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.123053:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000116384d8/0x268e1a602cc1154 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d6dd4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.123070:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f051c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280371200:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.123937:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294764672 00000100:00100000:3.0:1787827203.123944:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294764672 00000100:00100000:3.0:1787827203.123975:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 15 00000100:00100000:3.0:1787827203.123983:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3f04c4000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294764672:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.123991:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000116384d8/0x268e1a602cc1154 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x65:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d6dd4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.124026:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.124031:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3f04c4000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294764672:12345-192.168.203.124@tcp:104: Request processed in 49us (147us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.124037:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 15 00000100:00100000:3.0:1787827203.125862:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f062c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280371456:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.133134:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.133145:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294764928 00000100:00100000:3.0:1787827203.133150:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294764928 00000100:00100000:3.0:1787827203.133181:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 17 00000100:00100000:3.0:1787827203.133187:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3767050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294764928:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.133197:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009b0a1ac4/0x268e1a602cc1177 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x66:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d6e2f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.133233:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.133238:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3767050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294764928:12345-192.168.203.124@tcp:104: Request processed in 52us (158us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.133244:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 17 00000100:00100000:3.0:1787827203.139848:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f051e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280373504:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.140593:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000003d3c6afd/0x268e1a602cc119a lrc: 6/2,0 mode: --/PR res: [0x280000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.140604:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000003d3c6afd/0x268e1a602cc119a lrc: 6/2,0 mode: --/PR res: [0x280000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6e8a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.140612:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000003d3c6afd/0x268e1a602cc119a lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6e8a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.140619:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000003d3c6afd/0x268e1a602cc119a lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6e8a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.140627:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000003d3c6afd/0x268e1a602cc119a lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6e8a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.140639:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.140647:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000003d3c6afd/0x268e1a602cc119a lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6e8a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.140653:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000003d3c6afd/0x268e1a602cc119a lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6e8a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.140659:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000003d3c6afd/0x268e1a602cc119a lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6e8a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.140664:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000003d3c6afd/0x268e1a602cc119a lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d6e8a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.140674:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f051e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280373504:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.141547:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294765184 00000100:00100000:3.0:1787827203.141552:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294765184 00000100:00100000:3.0:1787827203.141581:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 19 00000100:00100000:3.0:1787827203.141587:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fc3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294765184:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.141595:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000003d3c6afd/0x268e1a602cc119a lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x66:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d6e8a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.141628:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.141634:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fc3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294765184:12345-192.168.203.124@tcp:104: Request processed in 47us (128us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.141647:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 19 00000100:00100000:3.0:1787827203.143359:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f062d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280373760:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.149835:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294765440 00000100:00100000:3.0:1787827203.149841:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294765440 00000100:00100000:3.0:1787827203.149870:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 21 00000100:00100000:3.0:1787827203.149875:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fe940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294765440:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.149900:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b209a86c/0x268e1a602cc11bd lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x67:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d6ee5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.149937:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.149942:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fe940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294765440:12345-192.168.203.124@tcp:104: Request processed in 67us (151us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.149947:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 21 00000100:00100000:3.0:1787827203.156733:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f051e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280375808:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.157455:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b6f4dc4f/0x268e1a602cc11e0 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.157466:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b6f4dc4f/0x268e1a602cc11e0 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6f40 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.157474:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b6f4dc4f/0x268e1a602cc11e0 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6f40 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.157482:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b6f4dc4f/0x268e1a602cc11e0 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6f40 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.157490:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b6f4dc4f/0x268e1a602cc11e0 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6f40 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.157497:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.157506:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b6f4dc4f/0x268e1a602cc11e0 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6f40 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.157511:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b6f4dc4f/0x268e1a602cc11e0 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6f40 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.157517:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b6f4dc4f/0x268e1a602cc11e0 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6f40 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.157524:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b6f4dc4f/0x268e1a602cc11e0 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d6f40 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.157534:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f051e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280375808:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.158388:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.158397:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294765696 00000100:00100000:3.0:1787827203.158401:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294765696 00000100:00100000:3.0:1787827203.158428:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 23 00000100:00100000:3.0:1787827203.158434:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3764450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294765696:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.158442:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b6f4dc4f/0x268e1a602cc11e0 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x67:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d6f40 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.158480:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.158484:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3764450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294765696:12345-192.168.203.124@tcp:104: Request processed in 51us (137us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.158490:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 23 00000100:00100000:3.0:1787827203.158644:0:10956:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1787827203.158654:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f062da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280376064:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.160067:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f062da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280376064:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.167022:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294765952 00000100:00100000:3.0:1787827203.167028:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294765952 00000100:00100000:3.0:1787827203.167058:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 25 00000100:00100000:3.0:1787827203.167066:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fcf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294765952:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.167074:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093995238/0x268e1a602cc1203 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x68:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d6f9b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.167123:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.167129:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fcf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294765952:12345-192.168.203.124@tcp:104: Request processed in 64us (145us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.167137:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 25 00000100:00100000:3.0:1787827203.173722:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f051e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280378112:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.174485:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1226 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.174497:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1226 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6ff6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.174507:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1226 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6ff6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.174519:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1226 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6ff6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.174530:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1226 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d6ff6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.174540:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.174553:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1226 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6ff6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.174563:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1226 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6ff6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.174574:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1226 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d6ff6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.174595:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1226 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d6ff6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.174620:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f051e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280378112:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.175550:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294766208 00000100:00100000:3.0:1787827203.175556:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294766208 00000100:00100000:3.0:1787827203.175592:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 27 00000100:00100000:3.0:1787827203.175603:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fe1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294766208:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.175613:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1226 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x68:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d6ff6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.175658:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.175666:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fe1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294766208:12345-192.168.203.124@tcp:104: Request processed in 65us (171us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.175676:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 27 00000100:00100000:3.0:1787827203.177349:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f062fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280378368:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.184028:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.184042:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294766464 00000100:00100000:3.0:1787827203.184047:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294766464 00000100:00100000:3.0:1787827203.184084:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 29 00000100:00100000:3.0:1787827203.184095:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3764850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294766464:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.184119:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000001f0ec90a/0x268e1a602cc1249 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x69:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7051 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.184168:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.184177:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3764850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294766464:12345-192.168.203.124@tcp:104: Request processed in 83us (203us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.184188:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 29 00000100:00100000:3.0:1787827203.191097:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0890b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280380416:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.192006:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d2f60288/0x268e1a602cc126c lrc: 6/2,0 mode: --/PR res: [0x280000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.192019:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d2f60288/0x268e1a602cc126c lrc: 6/2,0 mode: --/PR res: [0x280000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d70ac expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.192028:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d2f60288/0x268e1a602cc126c lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d70ac expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.192041:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d2f60288/0x268e1a602cc126c lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d70ac expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.192052:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d2f60288/0x268e1a602cc126c lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d70ac expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.192063:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.192077:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d2f60288/0x268e1a602cc126c lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d70ac expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.192087:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d2f60288/0x268e1a602cc126c lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d70ac expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.192095:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d2f60288/0x268e1a602cc126c lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d70ac expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.192102:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d2f60288/0x268e1a602cc126c lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d70ac expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.192128:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0890b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280380416:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.193168:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294766720 00000100:00100000:3.0:1787827203.193175:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294766720 00000100:00100000:3.0:1787827203.193213:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 31 00000100:00100000:3.0:1787827203.193223:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fda40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294766720:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.193234:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d2f60288/0x268e1a602cc126c lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x69:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d70ac expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.193280:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.193287:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fda40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294766720:12345-192.168.203.124@tcp:104: Request processed in 66us (183us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.193295:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 31 00000100:00100000:3.0:1787827203.193448:0:10956:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1787827203.193461:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f062d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280380672:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.196062:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f062d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280380672:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.201724:0:10955:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1787827203.201736:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0891680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280381568:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.202551:0:10955:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc128f lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.202562:0:10955:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc128f lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7107 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.202575:0:10955:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc128f lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d7107 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.202582:0:10955:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc128f lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7107 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.202597:0:10955:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc128f lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d7107 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.202604:0:10955:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.202611:0:10955:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc128f lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7107 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.202617:0:10955:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc128f lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7107 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.202625:0:10955:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc128f lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7107 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.202632:0:10955:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc128f lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d7107 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.202642:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0891680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280381568:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.203562:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294766976 00000100:00100000:3.0:1787827203.203568:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294766976 00000100:00100000:3.0:1787827203.203605:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 33 00000100:00100000:3.0:1787827203.203612:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fd680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294766976:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.203621:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc128f lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x6a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7107 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.203660:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.203668:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fd680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294766976:12345-192.168.203.124@tcp:104: Request processed in 56us (149us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.203678:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 33 00000100:00100000:3.0:1787827203.210066:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08903c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280382720:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.210787:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc12b2 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.210803:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc12b2 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7162 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.210815:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc12b2 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d7162 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.210826:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc12b2 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7162 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.210837:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc12b2 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d7162 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.210845:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.210857:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc12b2 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7162 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.210867:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc12b2 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7162 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.210877:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc12b2 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7162 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.210919:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc12b2 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d7162 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.210934:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d08903c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280382720:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.211823:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.211832:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294767232 00000100:00100000:3.0:1787827203.211836:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294767232 00000100:00100000:3.0:1787827203.211865:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 35 00000100:00100000:3.0:1787827203.211874:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3767450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294767232:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.211906:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc12b2 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x6a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7162 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.211942:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.211947:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3767450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294767232:12345-192.168.203.124@tcp:104: Request processed in 75us (168us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.211953:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 35 00000100:00100000:3.0:1787827203.213796:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f062e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280382976:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.220522:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294767488 00000100:00100000:3.0:1787827203.220528:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294767488 00000100:00100000:3.0:1787827203.220558:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 37 00000100:00100000:3.0:1787827203.220569:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fcb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294767488:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.220580:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093995238/0x268e1a602cc12d5 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x6b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d71bd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.220627:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.220635:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fcb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294767488:12345-192.168.203.124@tcp:104: Request processed in 67us (154us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.220644:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 37 00000100:00100000:3.0:1787827203.227130:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0891a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280385024:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.227992:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc12f8 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.228003:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc12f8 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7218 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.228011:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc12f8 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d7218 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.228018:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc12f8 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7218 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.228026:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc12f8 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d7218 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.228033:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.228042:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc12f8 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7218 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.228048:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc12f8 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7218 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.228053:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc12f8 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7218 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.228059:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc12f8 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d7218 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.228075:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0891a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280385024:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.229038:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294767744 00000100:00100000:3.0:1787827203.229043:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294767744 00000100:00100000:3.0:1787827203.229070:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 39 00000100:00100000:3.0:1787827203.229077:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22ffc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294767744:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.229085:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc12f8 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x6b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7218 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.229135:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.229140:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22ffc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294767744:12345-192.168.203.124@tcp:104: Request processed in 64us (143us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.229146:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 39 00000100:00100000:3.0:1787827203.230979:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f062e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280385280:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.237653:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.237663:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294768000 00000100:00100000:3.0:1787827203.237667:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294768000 00000100:00100000:3.0:1787827203.237698:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 41 00000100:00100000:3.0:1787827203.237709:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3766050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294768000:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.237720:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000002dea2600/0x268e1a602cc131b lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x6c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7273 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.237763:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.237769:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3766050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294768000:12345-192.168.203.124@tcp:104: Request processed in 64us (161us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.237776:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 41 00000100:00100000:3.0:1787827203.239565:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0891e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280386432:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.244667:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0892d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280387328:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.245411:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc133e lrc: 6/2,0 mode: --/PR res: [0x280000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.245427:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc133e lrc: 6/2,0 mode: --/PR res: [0x280000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d72ce expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.245440:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc133e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d72ce expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.245452:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc133e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d72ce expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.245464:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc133e lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d72ce expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.245471:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.245481:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc133e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d72ce expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.245487:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc133e lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d72ce expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.245492:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc133e lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d72ce expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.245498:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc133e lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d72ce expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.245509:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0892d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280387328:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.246395:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294768256 00000100:00100000:3.0:1787827203.246401:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294768256 00000100:00100000:3.0:1787827203.246437:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 43 00000100:00100000:3.0:1787827203.246447:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3f062d680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294768256:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.246459:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc133e lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x6c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d72ce expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.246502:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.246507:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3f062d680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294768256:12345-192.168.203.124@tcp:104: Request processed in 64us (148us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.246514:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 43 00000100:00100000:3.0:1787827203.248206:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f062fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280387584:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.253428:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0631680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280388480:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.254151:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000009fe1c17/0x268e1a602cc1361 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.254162:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000009fe1c17/0x268e1a602cc1361 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7329 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.254171:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000009fe1c17/0x268e1a602cc1361 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d7329 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.254178:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000009fe1c17/0x268e1a602cc1361 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7329 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.254190:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000009fe1c17/0x268e1a602cc1361 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d7329 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.254197:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.254206:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000009fe1c17/0x268e1a602cc1361 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7329 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.254212:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000009fe1c17/0x268e1a602cc1361 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7329 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.254217:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000009fe1c17/0x268e1a602cc1361 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7329 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.254223:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000009fe1c17/0x268e1a602cc1361 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d7329 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.254232:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0631680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280388480:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.255173:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294768512 00000100:00100000:3.0:1787827203.255178:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294768512 00000100:00100000:3.0:1787827203.255210:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 45 00000100:00100000:3.0:1787827203.255217:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3f062da40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294768512:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.255225:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000009fe1c17/0x268e1a602cc1361 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x6d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7329 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.255264:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.255269:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3f062da40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294768512:12345-192.168.203.124@tcp:104: Request processed in 53us (141us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.255274:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 45 00000100:00100000:3.0:1787827203.261957:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d06303c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280389632:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.262785:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc1384 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.262796:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc1384 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7384 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.262804:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc1384 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d7384 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.262811:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc1384 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7384 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.262819:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc1384 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d7384 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.262826:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.262834:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc1384 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7384 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.262844:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc1384 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7384 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.262850:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc1384 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7384 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.262855:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc1384 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d7384 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.262865:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d06303c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280389632:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.263874:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.263910:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294768768 00000100:00100000:3.0:1787827203.263917:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294768768 00000100:00100000:3.0:1787827203.263967:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 47 00000100:00100000:3.0:1787827203.263980:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3767050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294768768:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.263991:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc1384 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x6d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7384 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.264046:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.264054:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3767050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294768768:12345-192.168.203.124@tcp:104: Request processed in 78us (231us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.264063:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 47 00000100:00100000:3.0:1787827203.266017:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0630780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280389888:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.271238:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0630f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280390784:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.272035:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093995238/0x268e1a602cc13a7 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.272046:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093995238/0x268e1a602cc13a7 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d73df expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.272054:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093995238/0x268e1a602cc13a7 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d73df expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.272062:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093995238/0x268e1a602cc13a7 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d73df expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.272069:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093995238/0x268e1a602cc13a7 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d73df expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.272076:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.272084:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093995238/0x268e1a602cc13a7 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d73df expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.272090:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093995238/0x268e1a602cc13a7 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d73df expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.272095:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093995238/0x268e1a602cc13a7 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d73df expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.272119:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093995238/0x268e1a602cc13a7 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d73df expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.272137:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0630f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280390784:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.273119:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294769024 00000100:00100000:3.0:1787827203.273124:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294769024 00000100:00100000:3.0:1787827203.273154:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 49 00000100:00100000:3.0:1787827203.273160:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3f062d2c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294769024:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.273168:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093995238/0x268e1a602cc13a7 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x6e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d73df expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.273204:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.273209:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3f062d2c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294769024:12345-192.168.203.124@tcp:104: Request processed in 49us (139us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.273214:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 49 00000100:00100000:3.0:1787827203.280339:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0631e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280391936:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.281157:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000037fa3f21/0x268e1a602cc13ca lrc: 6/2,0 mode: --/PR res: [0x280000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.281168:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000037fa3f21/0x268e1a602cc13ca lrc: 6/2,0 mode: --/PR res: [0x280000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d743a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.281177:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000037fa3f21/0x268e1a602cc13ca lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d743a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.281185:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000037fa3f21/0x268e1a602cc13ca lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d743a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.281194:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000037fa3f21/0x268e1a602cc13ca lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d743a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.281200:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.281210:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000037fa3f21/0x268e1a602cc13ca lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d743a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.281216:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000037fa3f21/0x268e1a602cc13ca lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d743a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.281221:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000037fa3f21/0x268e1a602cc13ca lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d743a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.281226:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000037fa3f21/0x268e1a602cc13ca lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d743a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.281235:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0631e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280391936:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.282218:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294769280 00000100:00100000:3.0:1787827203.282223:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294769280 00000100:00100000:3.0:1787827203.282252:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 51 00000100:00100000:3.0:1787827203.282258:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3f062c3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294769280:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.282266:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000037fa3f21/0x268e1a602cc13ca lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x6e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d743a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.282303:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.282308:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3f062c3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294769280:12345-192.168.203.124@tcp:104: Request processed in 51us (135us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.282313:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 51 00000100:00100000:3.0:1787827203.289631:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d06321c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280393088:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.290399:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d1a50dcb/0x268e1a602cc13ed lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.290409:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d1a50dcb/0x268e1a602cc13ed lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7495 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.290417:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d1a50dcb/0x268e1a602cc13ed lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d7495 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.290425:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d1a50dcb/0x268e1a602cc13ed lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7495 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.290431:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d1a50dcb/0x268e1a602cc13ed lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d7495 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.290437:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.290446:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d1a50dcb/0x268e1a602cc13ed lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7495 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.290456:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d1a50dcb/0x268e1a602cc13ed lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7495 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.290466:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d1a50dcb/0x268e1a602cc13ed lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7495 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.290485:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d1a50dcb/0x268e1a602cc13ed lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d7495 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.290508:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d06321c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280393088:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.291420:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.291430:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294769536 00000100:00100000:3.0:1787827203.291433:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294769536 00000100:00100000:3.0:1787827203.291463:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 53 00000100:00100000:3.0:1787827203.291473:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3764850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294769536:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.291483:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d1a50dcb/0x268e1a602cc13ed lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x6f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7495 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.291527:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.291535:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3764850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294769536:12345-192.168.203.124@tcp:104: Request processed in 64us (156us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.291543:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 53 00000100:00100000:3.0:1787827203.297852:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0633480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280394240:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.299642:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294769792 00000100:00100000:3.0:1787827203.299647:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294769792 00000100:00100000:3.0:1787827203.299673:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 55 00000100:00100000:3.0:1787827203.299678:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3f062e580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294769792:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.299687:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000bb8b36bf/0x268e1a602cc1410 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x6f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d74f0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.299722:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.299727:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3f062e580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294769792:12345-192.168.203.124@tcp:104: Request processed in 49us (125us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.299732:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 55 00000100:00100000:3.0:1787827203.306351:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0631a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280395392:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.307115:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc1433 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.307128:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc1433 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d754b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.307136:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc1433 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d754b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.307143:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc1433 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d754b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.307150:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc1433 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d754b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.307157:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.307165:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc1433 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d754b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.307171:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc1433 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d754b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.307177:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc1433 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d754b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.307183:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc1433 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d754b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.307195:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0631a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280395392:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.308052:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294770048 00000100:00100000:3.0:1787827203.308057:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294770048 00000100:00100000:3.0:1787827203.308081:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 57 00000100:00100000:3.0:1787827203.308087:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fc780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294770048:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.308094:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc1433 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x70:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d754b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.308137:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.308142:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fc780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294770048:12345-192.168.203.124@tcp:104: Request processed in 57us (127us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.308151:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 57 00000100:00100000:3.0:1787827203.309791:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0631a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280395648:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.315595:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.315604:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294770304 00000100:00100000:3.0:1787827203.315607:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294770304 00000100:00100000:3.0:1787827203.315629:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 59 00000100:00100000:3.0:1787827203.315634:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3767450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294770304:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.315641:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000023d8b5f2/0x268e1a602cc1456 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x70:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d75a6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.315673:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.315681:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3767450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77209:x1874671294770304:12345-192.168.203.124@tcp:104: Request processed in 45us (123us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.315689:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 59 00000100:00100000:3.0:1787827203.323016:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294770560 00000100:00100000:3.0:1787827203.323021:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294770560 00000100:00100000:3.0:1787827203.323044:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 61 00000100:00100000:3.0:1787827203.323050:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fe940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294770560:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.323058:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006b3f8eff/0x268e1a602cc1479 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x71:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7601 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.323089:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.323094:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fe940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77208:x1874671294770560:12345-192.168.203.124@tcp:104: Request processed in 45us (123us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.323100:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 61 00000100:00100000:3.0:1787827203.330970:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294770816 00000100:00100000:3.0:1787827203.330976:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294770816 00000100:00100000:3.0:1787827203.331010:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 63 00000100:00100000:3.0:1787827203.331019:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fe580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294770816:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.331027:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000004f732e26/0x268e1a602cc149c lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x71:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d765c expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.331062:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.331067:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fe580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294770816:12345-192.168.203.124@tcp:104: Request processed in 50us (143us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.331072:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 63 00000100:00100000:3.0:1787827203.337462:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f062d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280400000:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.338298:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc14bf lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.338309:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc14bf lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d76b7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.338317:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc14bf lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d76b7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.338324:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc14bf lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d76b7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.338331:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc14bf lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d76b7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.338342:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.338351:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc14bf lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d76b7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.338356:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc14bf lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d76b7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.338363:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc14bf lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d76b7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.338369:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc14bf lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d76b7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.338380:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f062d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280400000:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.339308:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.339318:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294771200 00000100:00100000:3.0:1787827203.339322:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294771200 00000100:00100000:3.0:1787827203.339349:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 65 00000100:00100000:3.0:1787827203.339355:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3764c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294771200:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.339363:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc14bf lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x72:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d76b7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.339397:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.339402:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3764c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294771200:12345-192.168.203.124@tcp:104: Request processed in 48us (147us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.339408:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 65 00000100:00100000:3.0:1787827203.348435:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294771584 00000100:00100000:3.0:1787827203.348442:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294771584 00000100:00100000:3.0:1787827203.348482:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 67 00000100:00100000:3.0:1787827203.348494:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fc780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294771584:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.348506:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc14e2 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x72:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7712 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.348560:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.348568:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fc780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294771584:12345-192.168.203.124@tcp:104: Request processed in 76us (186us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.348578:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 67 00000100:00100000:3.0:1787827203.355835:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f062d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280402304:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.356680:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a3c3d29d/0x268e1a602cc1505 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.356696:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a3c3d29d/0x268e1a602cc1505 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d776d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.356707:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a3c3d29d/0x268e1a602cc1505 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d776d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.356717:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a3c3d29d/0x268e1a602cc1505 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d776d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.356734:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a3c3d29d/0x268e1a602cc1505 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d776d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.356743:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.356754:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a3c3d29d/0x268e1a602cc1505 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d776d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.356763:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a3c3d29d/0x268e1a602cc1505 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d776d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.356772:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a3c3d29d/0x268e1a602cc1505 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d776d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.356782:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a3c3d29d/0x268e1a602cc1505 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d776d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.356820:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f062d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280402304:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.357814:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294771840 00000100:00100000:3.0:1787827203.357820:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294771840 00000100:00100000:3.0:1787827203.357851:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 69 00000100:00100000:3.0:1787827203.357857:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fe1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294771840:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.357865:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a3c3d29d/0x268e1a602cc1505 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x73:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d776d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.357927:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.357933:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fe1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294771840:12345-192.168.203.124@tcp:104: Request processed in 76us (159us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.357938:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 69 00000100:00100000:3.0:1787827203.366743:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.366756:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294772096 00000100:00100000:3.0:1787827203.366762:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294772096 00000100:00100000:3.0:1787827203.366803:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 71 00000100:00100000:3.0:1787827203.366813:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3766050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294772096:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.366825:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002a3151d7/0x268e1a602cc1528 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x73:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d77c8 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.366869:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.366874:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3766050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294772096:12345-192.168.203.124@tcp:104: Request processed in 64us (193us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.366879:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 71 00000100:00100000:3.0:1787827203.368731:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0631680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280403712:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.375952:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294772352 00000100:00100000:3.0:1787827203.375958:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294772352 00000100:00100000:3.0:1787827203.375988:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 73 00000100:00100000:3.0:1787827203.375995:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fc000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294772352:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.376003:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082a8b406/0x268e1a602cc154b lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x74:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7823 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.376053:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.376061:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fc000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294772352:12345-192.168.203.124@tcp:104: Request processed in 66us (152us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.376070:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 73 00000100:00100000:3.0:1787827203.385722:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294772608 00000100:00100000:3.0:1787827203.385730:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294772608 00000100:00100000:3.0:1787827203.385772:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 75 00000100:00100000:3.0:1787827203.385784:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22ff840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294772608:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.385796:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d564540/0x268e1a602cc156e lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x74:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d787e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.385858:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.385867:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22ff840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294772608:12345-192.168.203.124@tcp:104: Request processed in 85us (206us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.385876:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 75 00000100:00100000:3.0:1787827203.393118:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e089f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280406912:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.393979:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000567b7b9e/0x268e1a602cc1591 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.393990:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000567b7b9e/0x268e1a602cc1591 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d78d9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.393998:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000567b7b9e/0x268e1a602cc1591 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d78d9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.394009:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000567b7b9e/0x268e1a602cc1591 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d78d9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.394018:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000567b7b9e/0x268e1a602cc1591 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d78d9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.394024:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.394032:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000567b7b9e/0x268e1a602cc1591 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d78d9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.394039:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000567b7b9e/0x268e1a602cc1591 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d78d9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.394044:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000567b7b9e/0x268e1a602cc1591 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d78d9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.394050:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000567b7b9e/0x268e1a602cc1591 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d78d9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.394065:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e089f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280406912:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.395026:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.395036:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294772864 00000100:00100000:3.0:1787827203.395040:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294772864 00000100:00100000:3.0:1787827203.395070:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 77 00000100:00100000:3.0:1787827203.395076:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3767050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294772864:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.395085:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000567b7b9e/0x268e1a602cc1591 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x75:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d78d9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.395134:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.395139:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3767050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294772864:12345-192.168.203.124@tcp:104: Request processed in 63us (158us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.395146:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 77 00000100:00100000:3.0:1787827203.404572:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294773120 00000100:00100000:3.0:1787827203.404577:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294773120 00000100:00100000:3.0:1787827203.404611:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 79 00000100:00100000:3.0:1787827203.404622:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22ffc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294773120:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.404634:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d564540/0x268e1a602cc15b4 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x75:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7934 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.404683:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.404691:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22ffc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294773120:12345-192.168.203.124@tcp:104: Request processed in 71us (200us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.404699:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 79 00000100:00100000:3.0:1787827203.414268:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294773376 00000100:00100000:3.0:1787827203.414274:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294773376 00000100:00100000:3.0:1787827203.414309:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 81 00000100:00100000:3.0:1787827203.414328:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fd2c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294773376:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.414339:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc15d7 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x76:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d798f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.414386:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.414395:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fd2c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294773376:12345-192.168.203.124@tcp:104: Request processed in 77us (171us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.414425:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 81 00000100:00100000:3.0:1787827203.416182:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0630b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280409472:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.424101:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.424126:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294773632 00000100:00100000:3.0:1787827203.424131:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294773632 00000100:00100000:3.0:1787827203.424164:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 83 00000100:00100000:3.0:1787827203.424175:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3767450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294773632:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.424186:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000008a98f536/0x268e1a602cc15fa lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x76:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d79ea expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.424229:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.424235:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3767450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294773632:12345-192.168.203.124@tcp:104: Request processed in 63us (186us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.424243:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 83 00000100:00100000:3.0:1787827203.433190:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294773888 00000100:00100000:3.0:1787827203.433195:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294773888 00000100:00100000:3.0:1787827203.433222:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 85 00000100:00100000:3.0:1787827203.433229:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22ff840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294773888:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.433237:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a064a45d/0x268e1a602cc161d lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x77:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7a45 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.433273:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.433278:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22ff840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294773888:12345-192.168.203.124@tcp:104: Request processed in 50us (131us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.433284:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 85 00000100:00100000:3.0:1787827203.440490:0:10957:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:3.0:1787827203.440504:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07c5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280412672:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.441388:0:10957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000280b8f71/0x268e1a602cc1640 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.441399:0:10957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000280b8f71/0x268e1a602cc1640 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7aa0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.441407:0:10957:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000280b8f71/0x268e1a602cc1640 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d7aa0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.441415:0:10957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000280b8f71/0x268e1a602cc1640 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7aa0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.441423:0:10957:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000280b8f71/0x268e1a602cc1640 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d7aa0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.441429:0:10957:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.441437:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000280b8f71/0x268e1a602cc1640 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7aa0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.441444:0:10957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000280b8f71/0x268e1a602cc1640 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7aa0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.441449:0:10957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000280b8f71/0x268e1a602cc1640 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7aa0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.441460:0:10957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000280b8f71/0x268e1a602cc1640 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d7aa0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.441474:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07c5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280412672:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.442547:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294774144 00000100:00100000:3.0:1787827203.442551:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294774144 00000100:00100000:3.0:1787827203.442576:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 87 00000100:00100000:3.0:1787827203.442581:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fc3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294774144:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.442589:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000280b8f71/0x268e1a602cc1640 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x77:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7aa0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.442623:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.442630:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fc3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294774144:12345-192.168.203.124@tcp:104: Request processed in 48us (122us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.442637:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 87 00000100:00100000:3.0:1787827203.451348:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07c4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280413824:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.452257:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a064a45d/0x268e1a602cc1663 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.452268:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a064a45d/0x268e1a602cc1663 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7afb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.452275:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a064a45d/0x268e1a602cc1663 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d7afb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.452283:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a064a45d/0x268e1a602cc1663 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7afb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.452290:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a064a45d/0x268e1a602cc1663 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d7afb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.452296:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.452304:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a064a45d/0x268e1a602cc1663 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7afb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.452310:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a064a45d/0x268e1a602cc1663 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7afb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.452315:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a064a45d/0x268e1a602cc1663 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7afb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.452321:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a064a45d/0x268e1a602cc1663 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d7afb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.452330:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07c4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280413824:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.453313:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.453326:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294774400 00000100:00100000:3.0:1787827203.453331:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294774400 00000100:00100000:3.0:1787827203.453374:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 89 00000100:00100000:3.0:1787827203.453382:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3764c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294774400:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.453391:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a064a45d/0x268e1a602cc1663 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x78:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7afb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.453429:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.453435:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3764c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294774400:12345-192.168.203.124@tcp:104: Request processed in 54us (178us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.453442:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 89 00000100:00100000:3.0:1787827203.453557:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0631680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280414080:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.455305:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0631680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280414080:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.462629:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294774656 00000100:00100000:3.0:1787827203.462636:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294774656 00000100:00100000:3.0:1787827203.462669:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 91 00000100:00100000:3.0:1787827203.462680:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fe1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294774656:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.462692:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b600453d/0x268e1a602cc1686 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x78:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7b56 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.462738:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.462746:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fe1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294774656:12345-192.168.203.124@tcp:104: Request processed in 68us (161us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.462755:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 91 00000100:00100000:3.0:1787827203.471079:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294774912 00000100:00100000:3.0:1787827203.471085:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294774912 00000100:00100000:3.0:1787827203.471130:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 93 00000100:00100000:3.0:1787827203.471137:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fe940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294774912:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.471146:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006ca7713c/0x268e1a602cc16a9 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x79:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7bb1 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.471181:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.471186:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fe940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294774912:12345-192.168.203.124@tcp:104: Request processed in 51us (159us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.471192:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 93 00000100:00100000:3.0:1787827203.479476:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.479485:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294775168 00000100:00100000:3.0:1787827203.479489:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294775168 00000100:00100000:3.0:1787827203.479520:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 95 00000100:00100000:3.0:1787827203.479526:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3766050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294775168:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.479533:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000015f7d211/0x268e1a602cc16cc lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x79:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7c0c expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.479575:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.479583:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3766050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294775168:12345-192.168.203.124@tcp:104: Request processed in 55us (151us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.479592:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 95 00000100:00100000:3.0:1787827203.486450:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07c6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280418432:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.487172:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000dab0575f/0x268e1a602cc16ef lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.487184:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000dab0575f/0x268e1a602cc16ef lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7c67 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.487198:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000dab0575f/0x268e1a602cc16ef lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d7c67 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.487205:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000dab0575f/0x268e1a602cc16ef lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7c67 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.487213:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000dab0575f/0x268e1a602cc16ef lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d7c67 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.487218:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.487229:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000dab0575f/0x268e1a602cc16ef lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7c67 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.487235:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000dab0575f/0x268e1a602cc16ef lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7c67 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.487239:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000dab0575f/0x268e1a602cc16ef lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7c67 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.487245:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000dab0575f/0x268e1a602cc16ef lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d7c67 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.487253:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07c6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280418432:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.488283:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294775424 00000100:00100000:3.0:1787827203.488289:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294775424 00000100:00100000:3.0:1787827203.488327:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 97 00000100:00100000:3.0:1787827203.488334:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22ff0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294775424:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.488342:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000dab0575f/0x268e1a602cc16ef lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x7a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7c67 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.488378:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.488383:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22ff0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294775424:12345-192.168.203.124@tcp:104: Request processed in 51us (144us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.488390:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 97 00000100:00100000:3.0:1787827203.488592:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0631680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280418688:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.490370:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0631680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280418688:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.497155:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294775680 00000100:00100000:3.0:1787827203.497162:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294775680 00000100:00100000:3.0:1787827203.497203:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 99 00000100:00100000:3.0:1787827203.497213:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fe1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294775680:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.497225:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc1712 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x7a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7cc2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.497272:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.497280:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fe1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294775680:12345-192.168.203.124@tcp:104: Request processed in 69us (183us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.497290:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 99 00000100:00100000:3.0:1787827203.504468:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07c6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280420736:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.505210:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1735 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.505221:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1735 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7d1d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.505231:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1735 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d7d1d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.505238:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1735 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7d1d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.505245:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1735 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d7d1d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.505252:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.505260:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1735 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7d1d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.505266:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1735 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7d1d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.505272:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1735 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7d1d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.505278:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1735 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d7d1d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.505289:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07c6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280420736:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.506276:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.506285:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294775936 00000100:00100000:3.0:1787827203.506290:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294775936 00000100:00100000:3.0:1787827203.506330:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 101 00000100:00100000:3.0:1787827203.506341:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3767050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294775936:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.506352:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1735 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x7b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7d1d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.506394:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.506402:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3767050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294775936:12345-192.168.203.124@tcp:104: Request processed in 63us (171us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.506412:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 101 00000100:00100000:3.0:1787827203.516183:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294776192 00000100:00100000:3.0:1787827203.516193:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294776192 00000100:00100000:3.0:1787827203.516231:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 103 00000100:00100000:3.0:1787827203.516239:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fed00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294776192:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.516253:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000fe5cc90c/0x268e1a602cc1758 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x7b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7d78 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.516305:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.516310:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fed00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294776192:12345-192.168.203.124@tcp:104: Request processed in 73us (174us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.516325:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 103 00010000:00010000:3.0:1787827203.524655:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009b82ab54/0x268e1a602cc177b lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.524674:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009b82ab54/0x268e1a602cc177b lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7dd3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.524691:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009b82ab54/0x268e1a602cc177b lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d7dd3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.524700:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009b82ab54/0x268e1a602cc177b lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7dd3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.524712:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009b82ab54/0x268e1a602cc177b lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d7dd3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.524721:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.524734:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009b82ab54/0x268e1a602cc177b lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7dd3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.524740:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009b82ab54/0x268e1a602cc177b lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7dd3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.524747:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009b82ab54/0x268e1a602cc177b lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7dd3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.524754:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009b82ab54/0x268e1a602cc177b lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d7dd3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.524769:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0631a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280423040:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.525820:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294776448 00000100:00100000:3.0:1787827203.525826:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294776448 00000100:00100000:3.0:1787827203.525866:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 105 00000100:00100000:3.0:1787827203.525878:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fcf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294776448:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.525931:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009b82ab54/0x268e1a602cc177b lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x7c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7dd3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.525990:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.525996:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fcf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294776448:12345-192.168.203.124@tcp:104: Request processed in 122us (219us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.526003:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 105 00000100:00100000:3.0:1787827203.528095:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0631a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280423296:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.536134:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.536145:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294776704 00000100:00100000:3.0:1787827203.536150:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294776704 00000100:00100000:3.0:1787827203.536184:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 107 00000100:00100000:3.0:1787827203.536192:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3767450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294776704:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.536201:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000326d4389/0x268e1a602cc179e lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x7c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7e2e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.536242:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.536247:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3767450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294776704:12345-192.168.203.124@tcp:104: Request processed in 57us (173us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.536256:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 107 00000100:00100000:3.0:1787827203.536481:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0631680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280424448:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.538255:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0631680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280424448:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.543560:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a26580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280425344:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.544356:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000047b9d5a6/0x268e1a602cc17c1 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.544366:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000047b9d5a6/0x268e1a602cc17c1 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7e89 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.544374:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000047b9d5a6/0x268e1a602cc17c1 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d7e89 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.544381:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000047b9d5a6/0x268e1a602cc17c1 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7e89 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.544389:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000047b9d5a6/0x268e1a602cc17c1 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d7e89 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.544395:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.544406:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000047b9d5a6/0x268e1a602cc17c1 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7e89 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.544420:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000047b9d5a6/0x268e1a602cc17c1 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7e89 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.544431:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000047b9d5a6/0x268e1a602cc17c1 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7e89 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.544447:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000047b9d5a6/0x268e1a602cc17c1 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d7e89 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.544470:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a26580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280425344:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.545406:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294776960 00000100:00100000:3.0:1787827203.545413:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294776960 00000100:00100000:3.0:1787827203.545453:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 109 00000100:00100000:3.0:1787827203.545461:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d0a24f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294776960:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.545470:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000047b9d5a6/0x268e1a602cc17c1 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x7d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7e89 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.545507:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.545512:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d0a24f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294776960:12345-192.168.203.124@tcp:104: Request processed in 53us (145us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.545518:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 109 00000100:00100000:3.0:1787827203.545740:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0631680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280425600:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.547565:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0631680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280425600:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.554930:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827203.555288:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a27840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280426752:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.563298:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280427648:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.564117:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1807 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.564129:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1807 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7f3f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.564137:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1807 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d7f3f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.564144:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1807 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7f3f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.564152:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1807 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d7f3f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.564158:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.564169:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1807 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7f3f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.564175:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1807 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7f3f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.564180:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1807 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7f3f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.564191:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000897f0868/0x268e1a602cc1807 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d7f3f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.564200:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a24000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280427648:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.575168:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827203.583460:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a27840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280429952:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.584234:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000374dc4c6/0x268e1a602cc184d lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.584246:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000374dc4c6/0x268e1a602cc184d lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7ff5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.584254:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000374dc4c6/0x268e1a602cc184d lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d7ff5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.584261:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000374dc4c6/0x268e1a602cc184d lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d7ff5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.584269:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000374dc4c6/0x268e1a602cc184d lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d7ff5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.584276:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.584285:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000374dc4c6/0x268e1a602cc184d lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7ff5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.584291:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000374dc4c6/0x268e1a602cc184d lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7ff5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.584296:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000374dc4c6/0x268e1a602cc184d lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d7ff5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.584302:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000374dc4c6/0x268e1a602cc184d lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d7ff5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.584312:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a27840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280429952:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.585205:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294777984 00000100:00100000:3.0:1787827203.585211:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294777984 00000100:00100000:3.0:1787827203.585248:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 117 00000100:00100000:3.0:1787827203.585258:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d0631680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294777984:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.585268:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000374dc4c6/0x268e1a602cc184d lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x7f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d7ff5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.585315:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.585323:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d0631680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294777984:12345-192.168.203.124@tcp:104: Request processed in 66us (162us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.585333:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 117 00000100:00100000:3.0:1787827203.586964:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a26580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280430208:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.593582:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827203.601873:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a24f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280432256:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.602575:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cf078f7b/0x268e1a602cc1893 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.602587:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cf078f7b/0x268e1a602cc1893 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d80ab expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.602595:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cf078f7b/0x268e1a602cc1893 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d80ab expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.602601:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cf078f7b/0x268e1a602cc1893 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d80ab expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.602608:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cf078f7b/0x268e1a602cc1893 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d80ab expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.602614:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.602624:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cf078f7b/0x268e1a602cc1893 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d80ab expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.602630:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cf078f7b/0x268e1a602cc1893 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d80ab expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.602635:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cf078f7b/0x268e1a602cc1893 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d80ab expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.602645:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cf078f7b/0x268e1a602cc1893 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d80ab expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.602656:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a24f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280432256:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.603629:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294778496 00000100:00100000:3.0:1787827203.603635:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294778496 00000100:00100000:3.0:1787827203.603662:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 121 00000100:00100000:3.0:1787827203.603667:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d0631680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294778496:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.603676:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cf078f7b/0x268e1a602cc1893 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x80:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d80ab expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.603709:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.603716:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d0631680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294778496:12345-192.168.203.124@tcp:104: Request processed in 48us (124us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.603724:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 121 00000100:00100000:3.0:1787827203.603816:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787827203.603873:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a24f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280432512:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.605396:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a24f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280432512:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.613323:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827203.615175:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280433664:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.619864:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0647840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280434560:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.620593:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc18d9 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.620604:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc18d9 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d8161 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.620612:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc18d9 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d8161 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.620619:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc18d9 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d8161 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.620626:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc18d9 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d8161 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.620637:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.620648:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc18d9 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d8161 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.620659:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc18d9 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d8161 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.620669:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc18d9 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d8161 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.620697:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc18d9 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d8161 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.620715:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0647840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280434560:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.621618:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.621628:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294779008 00000100:00100000:3.0:1787827203.621632:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294779008 00000100:00100000:3.0:1787827203.621666:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 125 00000100:00100000:3.0:1787827203.621676:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3f0474850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294779008:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.621686:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc18d9 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x81:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d8161 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.621720:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.621725:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3f0474850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294779008:12345-192.168.203.124@tcp:104: Request processed in 52us (148us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.621731:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 125 00000100:00100000:3.0:1787827203.632323:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827203.641679:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280436864:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.642576:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2329d40/0x268e1a602cc191f lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.642589:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2329d40/0x268e1a602cc191f lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d8217 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.642599:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2329d40/0x268e1a602cc191f lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d8217 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.642611:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2329d40/0x268e1a602cc191f lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d8217 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.642621:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2329d40/0x268e1a602cc191f lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d8217 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.642628:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.642636:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2329d40/0x268e1a602cc191f lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d8217 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.642642:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2329d40/0x268e1a602cc191f lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d8217 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.642650:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2329d40/0x268e1a602cc191f lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d8217 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.642660:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2329d40/0x268e1a602cc191f lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d8217 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.642685:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280436864:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.643806:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294779520 00000100:00100000:3.0:1787827203.643812:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294779520 00000100:00100000:3.0:1787827203.643863:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 129 00000100:00100000:3.0:1787827203.643875:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d06470c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294779520:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.643884:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2329d40/0x268e1a602cc191f lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x82:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d8217 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.643968:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.643975:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d06470c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294779520:12345-192.168.203.124@tcp:104: Request processed in 103us (212us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.643982:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 129 00000100:00100000:3.0:1787827203.654261:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.654272:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294779776 00000100:00100000:3.0:1787827203.654277:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294779776 00000100:00100000:3.0:1787827203.654314:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 131 00000100:00100000:3.0:1787827203.654324:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3766450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294779776:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.654337:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009b82ab54/0x268e1a602cc1942 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x82:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d8272 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.654383:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.654389:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3766450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294779776:12345-192.168.203.124@tcp:104: Request processed in 68us (174us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.654397:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 131 00000100:00100000:3.0:1787827203.663623:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280439168:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.664437:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc1965 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.664452:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc1965 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d82cd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.664479:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc1965 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d82cd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.664495:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc1965 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d82cd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.664508:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc1965 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d82cd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.664519:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.664534:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc1965 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d82cd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.664543:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc1965 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d82cd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.664562:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc1965 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d82cd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.664573:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc1965 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d82cd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.664599:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280439168:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.665604:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294780032 00000100:00100000:3.0:1787827203.665611:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294780032 00000100:00100000:3.0:1787827203.665653:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 133 00000100:00100000:3.0:1787827203.665662:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d0644b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294780032:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.665674:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc1965 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x83:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d82cd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.665731:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.665738:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d0644b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294780032:12345-192.168.203.124@tcp:104: Request processed in 77us (179us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.665749:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 133 00000100:00100000:3.0:1787827203.675277:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294780288 00000100:00100000:3.0:1787827203.675291:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294780288 00000100:00100000:3.0:1787827203.675333:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 135 00000100:00100000:3.0:1787827203.675344:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d0644000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294780288:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.675355:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000006cb182b/0x268e1a602cc1988 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x83:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d8328 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.675404:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.675412:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d0644000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294780288:12345-192.168.203.124@tcp:104: Request processed in 70us (178us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.675423:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 135 00000100:00100000:3.0:1787827203.685815:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280441472:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.686581:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc19ab lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.686600:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc19ab lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d8383 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.686608:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc19ab lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d8383 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.686615:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc19ab lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d8383 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.686624:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc19ab lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d8383 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.686631:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.686639:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc19ab lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d8383 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.686648:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc19ab lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d8383 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.686657:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc19ab lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d8383 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.686666:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc19ab lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d8383 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.686684:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280441472:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.687651:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.687660:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294780544 00000100:00100000:3.0:1787827203.687665:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294780544 00000100:00100000:3.0:1787827203.687701:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 137 00000100:00100000:3.0:1787827203.687710:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3766c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294780544:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.687721:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc19ab lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x84:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d8383 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.687763:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.687771:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3766c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294780544:12345-192.168.203.124@tcp:104: Request processed in 61us (166us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.687780:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 137 00000100:00100000:3.0:1787827203.698296:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294780800 00000100:00100000:3.0:1787827203.698308:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294780800 00000100:00100000:3.0:1787827203.698343:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 139 00000100:00100000:3.0:1787827203.698350:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d06461c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294780800:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.698360:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc19ce lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x84:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d83de expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.698398:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.698403:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d06461c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294780800:12345-192.168.203.124@tcp:104: Request processed in 54us (148us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.698410:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 139 00000100:00100000:3.0:1787827203.716994:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294781056 00000100:00100000:3.0:1787827203.717000:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294781056 00000100:00100000:3.0:1787827203.717032:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 141 00000100:00100000:3.0:1787827203.717039:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d0647c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294781056:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.717048:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000312928ed/0x268e1a602cc19f1 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x85:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d8439 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.717085:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.717089:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d0647c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294781056:12345-192.168.203.124@tcp:104: Request processed in 52us (136us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.717096:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 141 00000100:00100000:3.0:1787827203.726272:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.726281:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294781312 00000100:00100000:3.0:1787827203.726285:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294781312 00000100:00100000:3.0:1787827203.726327:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 143 00000100:00100000:3.0:1787827203.726335:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3765450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294781312:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.726343:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc1a14 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x85:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d8494 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.726378:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.726383:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3765450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294781312:12345-192.168.203.124@tcp:104: Request processed in 49us (155us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.726390:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 143 00000100:00100000:3.0:1787827203.726552:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07312c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280445184:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.728325:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07312c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280445184:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.733478:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f2895680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280446080:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.734220:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ae9c8f85/0x268e1a602cc1a37 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.734230:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ae9c8f85/0x268e1a602cc1a37 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d84ef expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.734238:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ae9c8f85/0x268e1a602cc1a37 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d84ef expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.734246:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ae9c8f85/0x268e1a602cc1a37 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d84ef expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.734255:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ae9c8f85/0x268e1a602cc1a37 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d84ef expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.734261:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.734271:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ae9c8f85/0x268e1a602cc1a37 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d84ef expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.734280:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ae9c8f85/0x268e1a602cc1a37 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d84ef expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.734289:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ae9c8f85/0x268e1a602cc1a37 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d84ef expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.734297:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ae9c8f85/0x268e1a602cc1a37 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d84ef expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.734310:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f2895680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280446080:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.735313:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294781568 00000100:00100000:3.0:1787827203.735319:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294781568 00000100:00100000:3.0:1787827203.735355:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 145 00000100:00100000:3.0:1787827203.735364:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d0645a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294781568:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.735375:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ae9c8f85/0x268e1a602cc1a37 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x86:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d84ef expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.735416:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.735424:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d0645a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294781568:12345-192.168.203.124@tcp:104: Request processed in 60us (149us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.735432:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 145 00000100:00100000:3.0:1787827203.745283:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294781824 00000100:00100000:3.0:1787827203.745289:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294781824 00000100:00100000:3.0:1787827203.745319:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 147 00000100:00100000:3.0:1787827203.745326:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fcf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294781824:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.745336:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc1a5a lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x86:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d854a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.745376:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.745384:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fcf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294781824:12345-192.168.203.124@tcp:104: Request processed in 57us (139us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.745393:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 147 00000100:00100000:3.0:1787827203.754398:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f2897480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280448384:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827203.755310:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c4650440/0x268e1a602cc1a7d lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.755323:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c4650440/0x268e1a602cc1a7d lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d85a5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.755337:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c4650440/0x268e1a602cc1a7d lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d85a5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.755349:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c4650440/0x268e1a602cc1a7d lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d85a5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827203.755363:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c4650440/0x268e1a602cc1a7d lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d85a5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827203.755375:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827203.755385:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c4650440/0x268e1a602cc1a7d lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d85a5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.755392:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c4650440/0x268e1a602cc1a7d lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d85a5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.755397:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c4650440/0x268e1a602cc1a7d lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d85a5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827203.755403:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c4650440/0x268e1a602cc1a7d lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d85a5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827203.755419:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f2897480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280448384:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.756540:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827203.756550:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294782080 00000100:00100000:3.0:1787827203.756555:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294782080 00000100:00100000:3.0:1787827203.756590:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 149 00000100:00100000:3.0:1787827203.756598:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3c3767050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294782080:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.756606:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c4650440/0x268e1a602cc1a7d lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x87:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d85a5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.756646:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.756651:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3c3767050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:82330:x1874671294782080:12345-192.168.203.124@tcp:104: Request processed in 54us (163us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.756659:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 149 00000100:00100000:3.0:1787827203.766350:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294782336 00000100:00100000:3.0:1787827203.766357:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294782336 00000100:00100000:3.0:1787827203.766391:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 151 00000100:00100000:3.0:1787827203.766400:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3f2895a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294782336:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.766409:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ac8cbf19/0x268e1a602cc1aa0 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x87:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d8600 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.766448:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.766454:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3f2895a40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77297:x1874671294782336:12345-192.168.203.124@tcp:104: Request processed in 56us (162us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.766460:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 151 00000100:00100000:3.0:1787827203.766602:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f2894780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280449792:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.768592:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f2894780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280449792:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827203.776513:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0647c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280450688:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827203.778476:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294782592 00000100:00100000:3.0:1787827203.778482:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294782592 00000100:00100000:3.0:1787827203.778516:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 153 00000100:00100000:3.0:1787827203.778523:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3f2896d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294782592:12345-192.168.203.124@tcp:104: 00010000:00010000:3.0:1787827203.778533:0:47645:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000122e40e5/0x268e1a602cc1ac3 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x88:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d865b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.778576:0:47645:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1787827203.778583:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3f2896d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77210:x1874671294782592:12345-192.168.203.124@tcp:104: Request processed in 59us (143us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827203.778595:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 153 00000100:00100000:3.0:1787827203.827070:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280453760:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.827090:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x54:0x0](ffff8ab3f1371148) 00000080:00200000:3.0:1787827203.827117:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub1, unhashed 1 00000080:00200000:3.0:1787827203.827137:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.827145:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.827153:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.827160:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.827170:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.827173:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.827175:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.827178:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.827181:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.827187:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.827193:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.827196:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.827199:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.827201:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.827202:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.827208:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub2, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.827210:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.827213:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.827216:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub2, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.827218:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub2, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.827236:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.827241:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e3ad00 x1874671280454016/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.827251:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e3ad00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280454016:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.827283:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.828808:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e3ad00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280454016:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.828829:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x55:0x0](ffff8ab3f1371988) 00000080:00200000:3.0:1787827203.828850:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub2, unhashed 1 00000080:00200000:3.0:1787827203.828874:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.828908:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.828921:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.828933:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.828941:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.828945:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.828949:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.828954:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.828957:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.828966:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.828976:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.828981:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.828985:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.828989:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.828991:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.829000:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub3, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.829003:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.829008:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.829011:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub3, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.829015:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub3, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.829042:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.829047:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e3a580 x1874671280454272/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.829058:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e3a580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280454272:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.829089:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.830095:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e3a580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280454272:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.830124:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x56:0x0](ffff8ab3f1372a08) 00000080:00200000:3.0:1787827203.830138:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub3, unhashed 1 00000080:00200000:3.0:1787827203.830154:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.830161:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.830168:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.830175:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.830180:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.830183:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.830185:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.830189:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.830191:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.830200:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.830207:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.830209:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.830212:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.830213:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.830215:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.830220:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub4, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.830223:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.830225:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.830228:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub4, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.830230:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub4, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.830247:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.830252:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e3bc00 x1874671280454528/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.830262:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e3bc00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280454528:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.830291:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.831508:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e3bc00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280454528:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.831524:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x57:0x0](ffff8ab3f13763c8) 00000080:00200000:3.0:1787827203.831536:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub4, unhashed 1 00000080:00200000:3.0:1787827203.831552:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.831559:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.831566:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.831573:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.831578:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.831581:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.831583:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.831586:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.831589:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.831594:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.831600:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.831603:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.831605:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.831607:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.831609:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.831614:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub5, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.831616:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.831619:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.831622:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub5, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.831624:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub5, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.831641:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.831646:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e3a1c0 x1874671280454784/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.831660:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e3a1c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280454784:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.831696:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.832719:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e3a1c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280454784:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.832739:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x58:0x0](ffff8ab3f1374b08) 00000080:00200000:3.0:1787827203.832755:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub5, unhashed 1 00000080:00200000:3.0:1787827203.832784:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.832793:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.832802:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.832812:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.832820:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.832824:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.832827:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.832831:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.832833:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.832839:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.832845:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.832848:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.832850:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.832853:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.832854:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.832860:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub6, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.832862:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.832865:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.832867:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub6, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.832869:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub6, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.832912:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.832919:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e383c0 x1874671280455040/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.832935:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e383c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280455040:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.832964:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.833869:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e383c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280455040:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.833914:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x59:0x0](ffff8ab3f13700c8) 00000080:00200000:3.0:1787827203.833933:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub6, unhashed 1 00000080:00200000:3.0:1787827203.833962:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.833972:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.833982:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.833993:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.834000:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.834004:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.834006:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.834010:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.834013:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.834019:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.834029:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.834034:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.834038:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.834041:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.834044:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.834052:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub7, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.834056:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.834060:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.834064:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub7, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.834068:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub7, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.834088:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.834094:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e3b840 x1874671280455296/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.834122:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e3b840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280455296:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.834158:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.835150:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e3b840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280455296:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.835167:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x5a:0x0](ffff8ab3f1377448) 00000080:00200000:3.0:1787827203.835179:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub7, unhashed 1 00000080:00200000:3.0:1787827203.835195:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.835201:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.835208:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.835215:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.835220:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.835223:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.835225:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.835228:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.835231:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.835239:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.835249:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.835253:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.835257:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.835260:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.835263:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.835271:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub8, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.835275:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.835279:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.835283:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub8, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.835287:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub8, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.835306:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.835311:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e392c0 x1874671280455552/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.835330:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e392c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280455552:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.835366:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.836464:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e392c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280455552:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.836478:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x5b:0x0](ffff8ab3f1375348) 00000080:00200000:3.0:1787827203.836490:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub8, unhashed 1 00000080:00200000:3.0:1787827203.836509:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.836518:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.836526:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.836536:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.836544:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.836549:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.836552:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.836557:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.836561:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.836569:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.836579:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.836584:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.836589:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.836592:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.836595:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.836604:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub9, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.836607:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.836612:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.836615:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub9, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.836618:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub9, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.836645:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.836652:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e39a40 x1874671280455808/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.836666:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e39a40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280455808:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.836702:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.837651:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e39a40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280455808:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.837666:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x5c:0x0](ffff8ab3f1375b88) 00000080:00200000:3.0:1787827203.837677:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub9, unhashed 1 00000080:00200000:3.0:1787827203.837693:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.837699:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.837706:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.837712:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.837717:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.837720:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.837722:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.837725:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.837727:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.837733:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.837739:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.837741:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.837743:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.837745:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.837746:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.837752:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub10, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.837754:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.837756:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.837759:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub10, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.837761:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub10, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.837778:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.837782:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e38780 x1874671280456064/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.837792:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e38780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280456064:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.837819:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.838838:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e38780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280456064:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.838852:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x5d:0x0](ffff8ab3f1370908) 00000080:00200000:3.0:1787827203.838863:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub10, unhashed 1 00000080:00200000:3.0:1787827203.838879:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.838916:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.838925:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.838936:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.838944:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.838949:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.838951:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.838955:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.838957:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.838963:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.838969:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.838972:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.838974:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.838976:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.838978:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.838984:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub11, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.838986:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.838988:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.838991:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub11, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.838993:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub11, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.839011:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.839015:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e3b480 x1874671280456320/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.839030:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e3b480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280456320:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.839057:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.840193:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e3b480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280456320:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.840207:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x5e:0x0](ffff8ab3f1373248) 00000080:00200000:3.0:1787827203.840218:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub11, unhashed 1 00000080:00200000:3.0:1787827203.840235:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.840244:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.840252:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.840261:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.840269:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.840274:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.840277:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.840281:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.840286:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.840292:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.840299:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.840301:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.840304:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.840306:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.840308:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.840314:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub12, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.840315:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.840320:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.840323:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub12, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.840326:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub12, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.840350:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.840357:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e3b0c0 x1874671280456576/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.840372:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e3b0c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280456576:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.840400:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.841362:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e3b0c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280456576:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.841379:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x5f:0x0](ffff8ab3f13721c8) 00000080:00200000:3.0:1787827203.841390:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub12, unhashed 1 00000080:00200000:3.0:1787827203.841405:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.841411:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.841417:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.841424:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.841428:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.841432:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.841433:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.841437:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.841440:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.841445:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.841451:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.841453:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.841455:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.841457:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.841459:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.841465:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub13, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.841467:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.841470:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.841472:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub13, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.841474:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub13, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.841490:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.841496:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e39e00 x1874671280456832/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.841510:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e39e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280456832:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.841546:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.842522:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e39e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280456832:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.842544:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x60:0x0](ffff8ab3f13742c8) 00000080:00200000:3.0:1787827203.842560:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub13, unhashed 1 00000080:00200000:3.0:1787827203.842577:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.842585:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.842592:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.842600:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.842605:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.842608:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.842610:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.842613:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.842616:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.842621:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.842628:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.842631:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.842633:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.842635:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.842637:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.842643:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub14, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.842645:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.842648:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.842650:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub14, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.842652:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub14, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.842670:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.842683:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e38000 x1874671280457088/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.842694:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e38000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280457088:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.842721:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.843666:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e38000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280457088:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.843682:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x61:0x0](ffff8ab3f1376c08) 00000080:00200000:3.0:1787827203.843693:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub14, unhashed 1 00000080:00200000:3.0:1787827203.843709:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.843715:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.843721:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.843728:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.843733:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.843736:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.843737:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.843741:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.843743:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.843749:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.843757:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.843761:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.843765:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.843768:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.843771:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.843778:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub15, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.843781:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.843785:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.843789:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub15, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.843792:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub15, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.843817:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.843824:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e39680 x1874671280457344/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.843838:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e39680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280457344:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.843877:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.844835:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e39680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280457344:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.844852:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x62:0x0](ffff8ab3f1373a88) 00000080:00200000:3.0:1787827203.844863:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub15, unhashed 1 00000080:00200000:3.0:1787827203.844878:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.844899:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.844905:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.844912:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.844917:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.844920:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.844922:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.844925:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.844927:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.844933:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.844939:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.844941:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.844944:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.844945:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.844947:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.844953:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub16, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.844957:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.844961:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.844964:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub16, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.844968:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub16, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.844993:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.845000:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e38f00 x1874671280457600/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.845015:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e38f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280457600:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.845047:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.846010:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e38f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280457600:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.846041:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x63:0x0](ffff8ab3d1ea80c8) 00000080:00200000:3.0:1787827203.846053:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub16, unhashed 1 00000080:00200000:3.0:1787827203.846069:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.846076:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.846082:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.846089:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.846094:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.846097:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.846099:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.846117:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.846120:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.846126:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.846132:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.846135:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.846137:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.846139:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.846141:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.846146:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub17, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.846148:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.846151:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.846154:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub17, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.846156:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub17, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.846174:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.846178:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e3a940 x1874671280457856/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.846188:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e3a940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280457856:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.846216:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.847350:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e3a940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280457856:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.847367:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x64:0x0](ffff8ab3d1eacb08) 00000080:00200000:3.0:1787827203.847443:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub17, unhashed 1 00000080:00200000:3.0:1787827203.847461:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.847468:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.847477:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.847489:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.847497:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.847502:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.847505:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.847510:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.847514:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.847523:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.847533:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.847537:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.847541:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.847543:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.847545:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.847552:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub18, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.847554:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.847557:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.847560:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub18, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.847562:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub18, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.847582:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.847588:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c6e38b40 x1874671280458112/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.847600:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e38b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280458112:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.847638:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.848507:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e38b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280458112:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.848526:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x65:0x0](ffff8ab3d1eae3c8) 00000080:00200000:3.0:1787827203.848540:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub18, unhashed 1 00000080:00200000:3.0:1787827203.848556:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.848563:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.848569:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.848584:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.848589:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.848592:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.848594:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.848598:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.848600:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.848609:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.848619:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.848623:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.848627:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.848630:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.848633:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.848642:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub19, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.848646:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.848650:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.848654:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub19, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.848657:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub19, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.848679:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.848684:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0631680 x1874671280458368/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.848695:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0631680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280458368:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.848723:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.849818:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0631680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280458368:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.849840:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x66:0x0](ffff8ab3d1eaaa08) 00000080:00200000:3.0:1787827203.849856:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub19, unhashed 1 00000080:00200000:3.0:1787827203.849877:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.849905:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.849916:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.849928:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.849937:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.849943:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.849946:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.849952:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.849956:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.849966:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.849977:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.849982:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.849987:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.849990:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.849993:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.850002:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub20, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.850005:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.850010:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.850014:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub20, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.850018:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub20, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.850043:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.850051:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f07c6d00 x1874671280458624/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.850066:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07c6d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280458624:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.850113:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.851085:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07c6d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280458624:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.851099:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x67:0x0](ffff8ab3d1ea9988) 00000080:00200000:3.0:1787827203.851127:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub20, unhashed 1 00000080:00200000:3.0:1787827203.851143:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.851149:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.851156:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.851163:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.851168:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.851171:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.851173:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.851176:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.851179:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.851184:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.851190:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.851193:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.851196:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.851198:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.851207:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.851214:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub21, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.851216:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.851219:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.851221:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub21, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.851223:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub21, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.851241:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.851246:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0531680 x1874671280458880/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.851255:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0531680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280458880:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.851283:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.852203:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0531680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280458880:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.852218:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x68:0x0](ffff8ab3d1ea9148) 00000080:00200000:3.0:1787827203.852230:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub21, unhashed 1 00000080:00200000:3.0:1787827203.852246:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.852252:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.852258:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.852265:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.852270:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.852272:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.852274:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.852278:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.852280:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.852285:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.852292:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.852295:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.852299:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.852301:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.852304:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.852312:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub22, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.852316:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.852320:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.852323:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub22, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.852327:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub22, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.852351:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.852358:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f04c61c0 x1874671280459136/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.852371:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04c61c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280459136:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.852406:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.853342:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04c61c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280459136:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.853363:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x69:0x0](ffff8ab3d1eadb88) 00000080:00200000:3.0:1787827203.853379:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub22, unhashed 1 00000080:00200000:3.0:1787827203.853399:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.853409:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.853418:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.853429:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.853438:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.853442:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.853446:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.853450:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.853455:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.853464:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.853474:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.853479:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.853484:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.853487:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.853490:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.853498:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub23, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.853502:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.853506:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.853510:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub23, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.853514:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub23, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.853537:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.853545:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f04c6940 x1874671280459392/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.853560:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04c6940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280459392:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.853595:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.854594:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04c6940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280459392:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.854608:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x6a:0x0](ffff8ab3d1ea8908) 00000080:00200000:3.0:1787827203.854618:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub23, unhashed 1 00000080:00200000:3.0:1787827203.854634:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.854641:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.854646:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.854653:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.854658:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.854661:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.854663:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.854666:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.854668:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.854683:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.854692:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.854695:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.854698:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.854700:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.854702:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.854707:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub24, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.854709:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.854712:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.854714:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub24, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.854716:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub24, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.854733:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.854737:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f04c4b40 x1874671280459648/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.854747:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04c4b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280459648:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.854773:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.855777:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04c4b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280459648:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.855797:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x6b:0x0](ffff8ab3d1eab248) 00000080:00200000:3.0:1787827203.855811:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub24, unhashed 1 00000080:00200000:3.0:1787827203.855831:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.855840:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.855849:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.855860:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.855868:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.855872:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.855875:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.855880:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.855906:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.855917:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.855928:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.855933:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.855937:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.855941:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.855944:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.855952:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub25, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.855956:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.855961:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.855965:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub25, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.855968:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub25, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.855991:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.855998:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f04c4000 x1874671280459904/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.856013:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04c4000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280459904:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.856049:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.857032:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04c4000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280459904:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.857048:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x6c:0x0](ffff8ab3d1eaa1c8) 00000080:00200000:3.0:1787827203.857060:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub25, unhashed 1 00000080:00200000:3.0:1787827203.857076:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.857083:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.857090:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.857098:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.857119:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.857123:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.857125:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.857128:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.857131:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.857136:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.857143:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.857146:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.857149:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.857151:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.857152:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.857158:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub26, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.857160:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.857163:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.857165:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub26, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.857168:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub26, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.857186:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.857191:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0b62d00 x1874671280460160/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.857201:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0b62d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280460160:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.857231:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.858281:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0b62d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280460160:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.858298:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x6d:0x0](ffff8ab3d1eac2c8) 00000080:00200000:3.0:1787827203.858309:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub26, unhashed 1 00000080:00200000:3.0:1787827203.858324:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.858331:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.858337:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.858344:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.858349:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.858351:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.858353:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.858356:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.858358:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.858364:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.858370:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.858372:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.858375:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.858377:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.858378:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.858383:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub27, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.858386:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.858388:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.858390:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub27, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.858392:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub27, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.858409:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.858414:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075de00 x1874671280460416/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.858424:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280460416:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.858451:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.859456:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280460416:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.859470:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x6e:0x0](ffff8ab3d1eaec08) 00000080:00200000:3.0:1787827203.859481:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub27, unhashed 1 00000080:00200000:3.0:1787827203.859496:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.859502:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.859508:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.859514:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.859519:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.859521:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.859523:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.859527:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.859529:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.859535:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.859541:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.859543:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.859545:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.859547:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.859549:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.859554:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub28, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.859556:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.859559:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.859561:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub28, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.859563:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub28, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.859579:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.859583:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075f840 x1874671280460672/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.859593:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280460672:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.859619:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.860489:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280460672:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.860503:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x6f:0x0](ffff8ab3d1eaba88) 00000080:00200000:3.0:1787827203.860513:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub28, unhashed 1 00000080:00200000:3.0:1787827203.860528:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.860534:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.860540:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.860546:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.860551:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.860554:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.860556:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.860558:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.860561:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.860566:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.860572:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.860574:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.860576:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.860578:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.860579:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.860584:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub29, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.860586:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.860589:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.860591:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub29, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.860593:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub29, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.860608:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.860613:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075c000 x1874671280460928/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.860623:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280460928:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.860648:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.861645:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280460928:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.861660:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x70:0x0](ffff8ab3d1ead348) 00000080:00200000:3.0:1787827203.861670:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub29, unhashed 1 00000080:00200000:3.0:1787827203.861685:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.861691:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.861696:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.861703:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.861708:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.861710:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.861712:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.861715:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.861717:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.861723:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.861728:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.861731:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.861733:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.861735:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.861737:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.861742:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub30, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.861744:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.861747:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.861749:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub30, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.861751:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub30, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.861766:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.861771:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075e940 x1874671280461184/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.861780:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280461184:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.861806:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.862783:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280461184:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.862802:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x71:0x0](ffff8ab3d1eaf448) 00000080:00200000:3.0:1787827203.862816:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub30, unhashed 1 00000080:00200000:3.0:1787827203.862835:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.862843:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.862851:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.862862:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.862869:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.862873:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.862876:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.862879:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.862882:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.862912:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.862919:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.862922:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.862925:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.862927:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.862929:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.862934:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub31, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.862936:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.862939:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.862941:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub31, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.862943:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub31, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.862960:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.862965:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075f0c0 x1874671280461440/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.862975:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280461440:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.863000:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.863999:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280461440:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.864031:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x72:0x0](ffff8ab3d1eddb88) 00000080:00200000:3.0:1787827203.864043:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub31, unhashed 1 00000080:00200000:3.0:1787827203.864057:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.864064:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.864070:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.864076:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.864081:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.864084:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.864086:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.864089:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.864091:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.864096:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.864116:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.864119:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.864122:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.864123:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.864125:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.864131:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub32, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.864133:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.864136:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.864138:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub32, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.864140:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub32, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.864157:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.864162:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075f480 x1874671280461696/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.864171:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280461696:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.864196:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.865251:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280461696:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.865266:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x73:0x0](ffff8ab3d1ed8908) 00000080:00200000:3.0:1787827203.865279:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub32, unhashed 1 00000080:00200000:3.0:1787827203.865296:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.865302:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.865309:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.865317:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.865322:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.865325:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.865327:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.865330:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.865332:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.865338:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.865344:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.865347:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.865349:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.865351:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.865353:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.865358:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub33, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.865360:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.865363:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.865366:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub33, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.865368:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub33, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.865384:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.865389:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075cf00 x1874671280461952/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.865399:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280461952:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.865428:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.866315:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280461952:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.866331:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x74:0x0](ffff8ab3d1edb248) 00000080:00200000:3.0:1787827203.866343:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub33, unhashed 1 00000080:00200000:3.0:1787827203.866359:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.866365:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.866372:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.866379:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.866383:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.866386:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.866388:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.866392:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.866394:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.866399:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.866405:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.866408:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.866410:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.866412:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.866414:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.866419:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub34, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.866421:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.866424:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.866426:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub34, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.866428:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub34, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.866444:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.866449:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075c780 x1874671280462208/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.866459:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280462208:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.866491:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.867429:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280462208:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.867444:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x75:0x0](ffff8ab3d1eda1c8) 00000080:00200000:3.0:1787827203.867455:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub34, unhashed 1 00000080:00200000:3.0:1787827203.867470:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.867476:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.867483:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.867490:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.867495:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.867499:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.867503:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.867507:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.867511:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.867519:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.867529:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.867534:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.867538:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.867542:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.867544:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.867552:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub35, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.867555:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.867558:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.867561:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub35, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.867563:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub35, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.867581:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.867586:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075c3c0 x1874671280462464/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.867596:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c3c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280462464:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.867623:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.868652:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c3c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280462464:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.868667:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x76:0x0](ffff8ab3d1edc2c8) 00000080:00200000:3.0:1787827203.868678:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub35, unhashed 1 00000080:00200000:3.0:1787827203.868692:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.868698:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.868704:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.868710:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.868715:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.868718:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.868720:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.868723:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.868725:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.868730:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.868736:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.868739:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.868742:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.868743:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.868745:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.868750:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub36, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.868752:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.868755:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.868757:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub36, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.868759:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub36, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.868775:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.868779:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075d2c0 x1874671280462720/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.868789:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075d2c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280462720:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.868816:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.869714:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075d2c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280462720:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.869728:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x77:0x0](ffff8ab3d1edec08) 00000080:00200000:3.0:1787827203.869739:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub36, unhashed 1 00000080:00200000:3.0:1787827203.869754:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.869760:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.869766:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.869772:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.869777:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.869780:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.869782:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.869785:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.869787:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.869792:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.869798:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.869801:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.869803:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.869804:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.869806:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.869811:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub37, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.869813:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.869816:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.869818:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub37, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.869820:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub37, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.869836:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.869840:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075fc00 x1874671280462976/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.869850:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075fc00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280462976:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.869881:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.870759:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075fc00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280462976:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.870772:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x78:0x0](ffff8ab3d1edba88) 00000080:00200000:3.0:1787827203.870783:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub37, unhashed 1 00000080:00200000:3.0:1787827203.870799:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.870805:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.870811:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.870818:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.870822:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.870825:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.870827:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.870830:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.870832:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.870838:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.870844:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.870846:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.870848:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.870850:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.870852:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.870857:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub38, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.870859:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.870862:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.870864:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub38, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.870866:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub38, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.870882:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.870902:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075d680 x1874671280463232/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.870913:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075d680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280463232:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.870939:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.871978:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075d680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280463232:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.871993:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x79:0x0](ffff8ab3d1edd348) 00000080:00200000:3.0:1787827203.872003:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub38, unhashed 1 00000080:00200000:3.0:1787827203.872018:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.872024:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.872030:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.872036:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.872041:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.872044:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.872045:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.872048:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.872050:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.872056:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.872062:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.872064:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.872066:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.872068:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.872070:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.872075:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub39, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.872077:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.872079:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.872082:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub39, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.872084:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub39, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.872099:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.872119:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075e1c0 x1874671280463488/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.872129:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e1c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280463488:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.872155:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.873025:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e1c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280463488:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.873039:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x7a:0x0](ffff8ab3d1edf448) 00000080:00200000:3.0:1787827203.873050:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub39, unhashed 1 00000080:00200000:3.0:1787827203.873065:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.873071:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.873077:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.873083:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.873088:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.873090:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.873092:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.873096:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.873098:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.873118:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.873125:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.873128:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.873130:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.873132:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.873134:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.873141:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub40, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.873143:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.873146:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.873148:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub40, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.873150:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub40, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.873167:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.873171:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075e580 x1874671280463744/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.873181:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280463744:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.873208:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.874047:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280463744:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.874061:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x7b:0x0](ffff8ab3d1ed80c8) 00000080:00200000:3.0:1787827203.874072:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub40, unhashed 1 00000080:00200000:3.0:1787827203.874087:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.874093:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.874113:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.874121:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.874127:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.874130:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.874132:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.874135:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.874138:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.874144:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.874150:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.874153:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.874156:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.874157:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.874159:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.874165:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub41, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.874168:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.874170:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.874173:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub41, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.874175:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub41, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.874206:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.874211:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075cb40 x1874671280464000/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.874221:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280464000:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.874246:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.875265:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280464000:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.875279:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x7c:0x0](ffff8ab3d1edcb08) 00000080:00200000:3.0:1787827203.875291:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub41, unhashed 1 00000080:00200000:3.0:1787827203.875307:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.875313:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.875319:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.875325:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.875330:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.875333:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.875335:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.875338:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.875340:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.875345:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.875351:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.875353:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.875356:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.875357:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.875359:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.875365:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub42, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.875367:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.875369:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.875372:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub42, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.875374:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub42, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.875389:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.875394:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075da40 x1874671280464256/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.875403:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280464256:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.875429:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.876325:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280464256:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.876339:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x7d:0x0](ffff8ab3d1ede3c8) 00000080:00200000:3.0:1787827203.876350:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub42, unhashed 1 00000080:00200000:3.0:1787827203.876364:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.876371:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.876377:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.876383:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.876388:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.876391:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.876393:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.876396:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.876399:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.876404:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.876410:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.876412:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.876415:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.876416:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.876418:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.876423:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub43, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.876425:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.876428:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.876430:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub43, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.876432:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub43, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.876448:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.876452:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f075ed00 x1874671280464512/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.876462:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280464512:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.876488:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.877386:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280464512:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.877401:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x7e:0x0](ffff8ab3d1edaa08) 00000080:00200000:3.0:1787827203.877412:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub43, unhashed 1 00000080:00200000:3.0:1787827203.877426:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.877433:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.877439:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.877445:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.877450:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.877453:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.877454:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.877457:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.877459:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.877465:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.877471:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.877473:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.877475:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.877477:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.877478:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.877483:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub44, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.877485:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.877488:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.877490:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub44, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.877492:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub44, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.877512:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.877519:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aaf480 x1874671280464768/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.877532:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aaf480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280464768:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.877566:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.878596:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aaf480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280464768:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.878610:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x7f:0x0](ffff8ab3d1ed9988) 00000080:00200000:3.0:1787827203.878620:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub44, unhashed 1 00000080:00200000:3.0:1787827203.878635:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.878641:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.878647:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.878653:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.878658:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.878661:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.878663:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.878666:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.878668:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.878673:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.878678:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.878681:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.878683:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.878685:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.878686:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.878692:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub45, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.878693:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.878696:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.878698:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub45, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.878700:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub45, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.878715:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.878720:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aada40 x1874671280465024/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.878729:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aada40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280465024:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.878755:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.879643:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aada40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280465024:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.879658:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x80:0x0](ffff8ab3d1ed9148) 00000080:00200000:3.0:1787827203.879668:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub45, unhashed 1 00000080:00200000:3.0:1787827203.879684:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.879690:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.879696:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.879702:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.879707:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.879710:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.879712:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.879715:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.879718:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.879723:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.879729:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.879731:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.879733:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.879735:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.879737:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.879742:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub46, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.879744:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.879746:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.879748:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub46, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.879750:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub46, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.879766:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.879771:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aac780 x1874671280465280/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.879780:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aac780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280465280:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.879807:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.880766:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aac780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280465280:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.880791:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x81:0x0](ffff8ab3d1fa2a08) 00000080:00200000:3.0:1787827203.880803:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub46, unhashed 1 00000080:00200000:3.0:1787827203.880817:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.880824:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.880832:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.880842:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.880849:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.880853:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.880856:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.880860:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.880865:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.880874:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.880903:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.880907:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.880910:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.880912:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.880913:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.880919:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub47, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.880921:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.880924:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.880927:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub47, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.880929:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub47, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.880947:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.880952:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aaed00 x1874671280465536/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.880962:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aaed00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280465536:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.880988:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.881957:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aaed00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280465536:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.881971:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x82:0x0](ffff8ab3d1fa1988) 00000080:00200000:3.0:1787827203.881982:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub47, unhashed 1 00000080:00200000:3.0:1787827203.881997:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.882003:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.882008:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.882015:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.882020:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.882023:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.882024:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.882028:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.882030:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.882035:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.882041:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.882044:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.882046:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.882048:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.882050:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.882055:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub48, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.882057:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.882059:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.882061:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub48, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.882063:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub48, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.882079:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.882083:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aae580 x1874671280465792/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.882093:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aae580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280465792:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.882134:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.882994:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aae580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280465792:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.883008:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x83:0x0](ffff8ab3d1fa1148) 00000080:00200000:3.0:1787827203.883019:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub48, unhashed 1 00000080:00200000:3.0:1787827203.883034:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.883040:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.883046:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.883052:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.883057:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.883060:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.883062:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.883065:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.883067:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.883073:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.883079:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.883081:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.883083:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.883085:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.883087:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.883092:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub49, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.883094:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.883097:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.883114:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub49, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.883117:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub49, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.883144:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.883149:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aad680 x1874671280466048/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.883159:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aad680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280466048:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.883185:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.884212:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aad680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280466048:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.884227:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x84:0x0](ffff8ab3d1fa5b88) 00000080:00200000:3.0:1787827203.884238:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub49, unhashed 1 00000080:00200000:3.0:1787827203.884253:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.884259:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.884265:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.884272:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.884277:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.884280:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.884282:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.884284:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.884287:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.884292:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.884299:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.884301:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.884303:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.884305:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.884307:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.884312:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub50, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.884314:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.884316:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.884319:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub50, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.884321:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub50, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.884336:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.884341:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aac3c0 x1874671280466304/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.884350:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aac3c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280466304:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.884378:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.885329:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aac3c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280466304:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.885343:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x85:0x0](ffff8ab3d1fa0908) 00000080:00200000:3.0:1787827203.885354:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub50, unhashed 1 00000080:00200000:3.0:1787827203.885369:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.885375:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.885381:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.885387:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.885392:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.885395:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.885397:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.885399:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.885401:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.885406:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.885412:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.885415:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.885417:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.885419:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.885421:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.885426:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub51, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.885428:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.885431:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.885433:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub51, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.885435:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub51, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.885450:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.885454:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aaf840 x1874671280466560/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.885464:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aaf840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280466560:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.885489:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.886390:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aaf840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280466560:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.886404:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x86:0x0](ffff8ab3d1fa3248) 00000080:00200000:3.0:1787827203.886414:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub51, unhashed 1 00000080:00200000:3.0:1787827203.886429:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.886435:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.886441:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.886448:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.886453:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.886455:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.886457:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.886460:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.886463:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.886468:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.886474:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.886476:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.886479:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.886480:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.886482:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.886486:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub52, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.886488:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.886491:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.886493:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub52, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.886496:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub52, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.886519:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.886526:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aac000 x1874671280466816/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.886540:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aac000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280466816:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.886568:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.887484:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aac000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280466816:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.887500:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x87:0x0](ffff8ab3d1fa21c8) 00000080:00200000:3.0:1787827203.887515:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub52, unhashed 1 00000080:00200000:3.0:1787827203.887536:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.887545:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.887553:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.887560:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.887565:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.887568:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.887570:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.887573:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.887575:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.887580:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.887586:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.887589:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.887591:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.887593:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.887595:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.887600:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub53, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.887602:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.887605:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.887607:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub53, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.887609:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub53, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.887625:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.887630:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aafc00 x1874671280467072/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.887639:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aafc00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280467072:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.887667:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.888584:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aafc00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280467072:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.888597:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x88:0x0](ffff8ab3d1fa42c8) 00000080:00200000:3.0:1787827203.888607:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub53, unhashed 1 00000080:00200000:3.0:1787827203.888622:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.888628:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.888634:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.888641:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.888646:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.888648:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.888650:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.888653:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.888655:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.888660:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.888666:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.888668:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.888671:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.888673:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.888674:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.888679:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub54, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.888682:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.888684:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.888686:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub54, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.888688:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub54, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.888704:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.888708:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aacb40 x1874671280467328/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.888717:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aacb40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280467328:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.888743:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.889661:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aacb40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280467328:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.889676:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x89:0x0](ffff8ab3d1fa6c08) 00000080:00200000:3.0:1787827203.889686:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub54, unhashed 1 00000080:00200000:3.0:1787827203.889701:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.889707:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.889713:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.889720:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.889724:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.889727:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.889729:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.889732:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.889734:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.889740:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.889746:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.889748:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.889750:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.889752:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.889754:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.889759:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub55, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.889761:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.889763:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.889766:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub55, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.889768:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub55, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.889783:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.889788:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aad2c0 x1874671280467584/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.889797:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aad2c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280467584:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.889824:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.890760:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aad2c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280467584:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.890774:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x8a:0x0](ffff8ab3d1fa3a88) 00000080:00200000:3.0:1787827203.890785:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub55, unhashed 1 00000080:00200000:3.0:1787827203.890800:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.890807:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.890812:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.890819:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.890824:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.890827:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.890828:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.890831:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.890834:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.890839:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.890845:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.890848:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.890850:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.890852:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.890853:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.890858:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub56, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.890860:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.890863:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.890865:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub56, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.890867:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub56, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.890882:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.890904:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aade00 x1874671280467840/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.890914:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aade00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280467840:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.890940:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.891882:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aade00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280467840:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.891913:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x8b:0x0](ffff8ab3d1fa5348) 00000080:00200000:3.0:1787827203.891926:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub56, unhashed 1 00000080:00200000:3.0:1787827203.891943:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.891949:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.891955:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.891961:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.891966:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.891969:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.891971:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.891974:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.891976:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.891981:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.891987:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.891989:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.891992:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.891994:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.891996:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.892000:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub57, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.892002:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.892005:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.892007:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub57, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.892009:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub57, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.892034:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.892038:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aae1c0 x1874671280468096/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.892048:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aae1c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280468096:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.892074:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.892953:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aae1c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280468096:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.892967:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x8c:0x0](ffff8ab3d1fa7448) 00000080:00200000:3.0:1787827203.892978:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub57, unhashed 1 00000080:00200000:3.0:1787827203.892992:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.892998:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.893005:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.893011:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.893015:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.893018:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.893020:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.893023:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.893026:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.893031:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.893037:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.893039:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.893042:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.893043:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.893045:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.893050:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub58, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.893052:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.893054:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.893057:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub58, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.893059:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub58, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.893074:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.893078:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aaf0c0 x1874671280468352/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.893088:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aaf0c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280468352:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.893130:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.894070:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aaf0c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280468352:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.894083:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x8d:0x0](ffff8ab3d1fa00c8) 00000080:00200000:3.0:1787827203.894094:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub58, unhashed 1 00000080:00200000:3.0:1787827203.894125:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.894133:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.894139:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.894145:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.894150:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.894153:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.894155:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.894158:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.894160:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.894166:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.894172:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.894174:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.894176:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.894178:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.894180:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.894184:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub59, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.894187:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.894189:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.894192:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub59, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.894194:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub59, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.894209:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.894214:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aae940 x1874671280468608/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.894223:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aae940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280468608:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.894249:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.895335:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aae940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280468608:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.895349:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x8e:0x0](ffff8ab3d1fa4b08) 00000080:00200000:3.0:1787827203.895360:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub59, unhashed 1 00000080:00200000:3.0:1787827203.895375:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.895381:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.895387:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.895393:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.895398:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.895401:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.895402:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.895405:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.895407:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.895413:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.895418:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.895421:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.895423:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.895425:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.895427:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.895432:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub60, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.895434:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.895436:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.895439:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub60, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.895441:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub60, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.895455:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.895460:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aacf00 x1874671280468864/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.895469:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aacf00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280468864:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.895496:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.896338:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aacf00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280468864:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.896352:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x8f:0x0](ffff8ab3d1fa63c8) 00000080:00200000:3.0:1787827203.896362:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub60, unhashed 1 00000080:00200000:3.0:1787827203.896378:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.896384:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.896390:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.896397:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.896401:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.896404:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.896406:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.896409:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.896412:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.896417:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.896423:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.896425:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.896428:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.896429:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.896431:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.896436:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub61, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.896438:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.896441:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.896443:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub61, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.896445:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub61, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.896461:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.896466:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af8f00 x1874671280469120/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.896476:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af8f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280469120:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.896503:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.897379:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af8f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280469120:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.897404:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x90:0x0](ffff8ab3d1ef00c8) 00000080:00200000:3.0:1787827203.897418:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub61, unhashed 1 00000080:00200000:3.0:1787827203.897432:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.897439:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.897445:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.897452:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.897457:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.897460:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.897462:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.897464:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.897467:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.897472:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.897478:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.897481:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.897483:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.897485:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.897486:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.897491:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub62, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.897493:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.897497:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.897500:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub62, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.897503:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub62, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.897527:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.897534:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af92c0 x1874671280469376/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.897548:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af92c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280469376:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.897575:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.898456:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af92c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280469376:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.898469:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x91:0x0](ffff8ab3d1ef4b08) 00000080:00200000:3.0:1787827203.898480:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub62, unhashed 1 00000080:00200000:3.0:1787827203.898495:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.898504:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.898512:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.898522:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.898529:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.898533:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.898536:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.898541:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.898544:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.898553:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.898562:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.898567:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.898571:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.898573:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.898575:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.898580:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub63, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.898582:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.898585:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.898588:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub63, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.898589:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub63, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.898607:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.898611:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afa580 x1874671280469632/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.898621:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afa580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280469632:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.898648:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.899521:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afa580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280469632:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.899541:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x92:0x0](ffff8ab3d1ef63c8) 00000080:00200000:3.0:1787827203.899555:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub63, unhashed 1 00000080:00200000:3.0:1787827203.899571:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.899577:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.899583:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.899590:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.899594:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.899597:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.899599:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.899602:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.899605:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.899610:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.899616:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.899618:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.899620:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.899622:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.899624:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.899629:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub64, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.899631:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.899633:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.899636:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub64, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.899638:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub64, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.899653:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.899658:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afb0c0 x1874671280469888/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.899668:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afb0c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280469888:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.899694:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.900559:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afb0c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280469888:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.900574:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x93:0x0](ffff8ab3d1ef2a08) 00000080:00200000:3.0:1787827203.900585:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub64, unhashed 1 00000080:00200000:3.0:1787827203.900600:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.900606:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.900612:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.900619:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.900624:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.900626:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.900628:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.900631:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.900633:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.900639:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.900645:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.900647:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.900650:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.900651:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.900653:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.900657:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub65, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.900659:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.900662:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.900664:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub65, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.900667:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub65, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.900691:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.900696:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af9a40 x1874671280470144/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.900705:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af9a40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280470144:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.900732:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.901839:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af9a40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280470144:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.901852:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x94:0x0](ffff8ab3d1ef1988) 00000080:00200000:3.0:1787827203.901864:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub65, unhashed 1 00000080:00200000:3.0:1787827203.901879:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.901903:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.901909:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.901916:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.901921:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.901924:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.901926:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.901929:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.901931:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.901936:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.901942:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.901944:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.901947:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.901949:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.901951:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.901956:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub66, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.901958:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.901961:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.901963:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub66, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.901965:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub66, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.901980:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.901985:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af8780 x1874671280470400/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.901994:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af8780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280470400:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.902020:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.902904:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af8780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280470400:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.902919:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x95:0x0](ffff8ab3d1ef1148) 00000080:00200000:3.0:1787827203.902930:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub66, unhashed 1 00000080:00200000:3.0:1787827203.902945:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.902951:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.902957:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.902963:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.902968:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.902971:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.902973:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.902976:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.902978:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.902984:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.902990:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.902992:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.902994:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.902996:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.902997:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.903002:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub67, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.903004:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.903007:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.903009:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub67, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.903011:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub67, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.903027:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.903031:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af9e00 x1874671280470656/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.903041:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af9e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280470656:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.903067:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.903960:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af9e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280470656:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.903974:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x96:0x0](ffff8ab3d1ef5b88) 00000080:00200000:3.0:1787827203.903985:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub67, unhashed 1 00000080:00200000:3.0:1787827203.904000:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.904007:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.904013:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.904020:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.904024:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.904027:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.904029:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.904032:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.904034:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.904040:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.904046:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.904048:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.904050:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.904052:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.904053:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.904058:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub68, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.904060:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.904063:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.904065:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub68, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.904068:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub68, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.904083:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.904088:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afb840 x1874671280470912/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.904097:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afb840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280470912:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.904139:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.905145:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afb840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280470912:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.905159:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x97:0x0](ffff8ab3d1ef0908) 00000080:00200000:3.0:1787827203.905170:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub68, unhashed 1 00000080:00200000:3.0:1787827203.905185:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.905191:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.905197:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.905203:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.905208:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.905211:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.905213:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.905216:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.905218:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.905223:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.905229:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.905231:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.905234:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.905236:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.905238:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.905242:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub69, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.905244:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.905247:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.905249:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub69, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.905251:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub69, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.905266:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.905270:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afb480 x1874671280471168/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.905279:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afb480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280471168:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.905305:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.906248:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afb480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280471168:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.906262:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x98:0x0](ffff8ab3d1ef3248) 00000080:00200000:3.0:1787827203.906273:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub69, unhashed 1 00000080:00200000:3.0:1787827203.906288:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.906294:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.906300:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.906307:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.906311:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.906314:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.906316:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.906320:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.906322:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.906327:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.906334:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.906336:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.906338:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.906340:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.906342:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.906348:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub70, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.906350:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.906353:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.906355:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub70, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.906357:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub70, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.906373:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.906377:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afad00 x1874671280471424/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.906387:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afad00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280471424:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.906412:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.907345:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afad00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280471424:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.907360:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x99:0x0](ffff8ab3d1ef21c8) 00000080:00200000:3.0:1787827203.907371:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub70, unhashed 1 00000080:00200000:3.0:1787827203.907386:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.907393:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.907399:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.907406:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.907410:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.907413:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.907415:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.907418:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.907420:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.907426:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.907432:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.907434:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.907436:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.907438:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.907440:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.907446:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub71, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.907448:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.907451:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.907454:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub71, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.907456:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub71, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.907471:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.907476:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af8000 x1874671280471680/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.907486:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af8000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280471680:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.907516:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.908517:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af8000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280471680:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.908536:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x9a:0x0](ffff8ab3d1ef42c8) 00000080:00200000:3.0:1787827203.908552:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub71, unhashed 1 00000080:00200000:3.0:1787827203.908570:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.908577:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.908583:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.908590:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.908594:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.908598:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.908599:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.908602:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.908605:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.908610:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.908616:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.908618:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.908621:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.908622:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.908624:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.908630:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub72, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.908632:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.908635:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.908637:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub72, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.908639:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub72, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.908655:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.908659:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afa940 x1874671280471936/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.908669:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afa940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280471936:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.908695:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.909582:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afa940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280471936:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.909596:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x9b:0x0](ffff8ab3d1ef6c08) 00000080:00200000:3.0:1787827203.909607:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub72, unhashed 1 00000080:00200000:3.0:1787827203.909626:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.909636:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.909644:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.909653:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.909659:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.909663:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.909666:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.909670:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.909673:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.909680:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.909689:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.909693:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.909697:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.909699:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.909702:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.909709:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub73, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.909713:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.909717:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.909721:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub73, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.909724:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub73, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.909755:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.909760:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af83c0 x1874671280472192/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.909770:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af83c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280472192:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.909796:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.910676:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af83c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280472192:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.910690:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x9c:0x0](ffff8ab3d1ef3a88) 00000080:00200000:3.0:1787827203.910701:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub73, unhashed 1 00000080:00200000:3.0:1787827203.910715:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.910722:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.910728:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.910734:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.910739:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.910742:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.910744:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.910746:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.910749:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.910754:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.910760:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.910762:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.910765:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.910766:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.910768:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.910773:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub74, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.910775:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.910777:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.910780:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub74, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.910782:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub74, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.910797:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.910802:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afa1c0 x1874671280472448/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.910811:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afa1c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280472448:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.910838:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.911804:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afa1c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280472448:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.911817:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x9d:0x0](ffff8ab3d1ef5348) 00000080:00200000:3.0:1787827203.911827:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub74, unhashed 1 00000080:00200000:3.0:1787827203.911842:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.911848:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.911853:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.911860:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.911864:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.911867:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.911869:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.911872:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.911874:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.911879:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.911901:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.911904:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.911908:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.911910:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.911913:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.911919:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub75, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.911921:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.911925:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.911928:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub75, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.911930:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub75, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.911947:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.911951:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af8b40 x1874671280472704/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.911961:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af8b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280472704:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.911987:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.912943:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af8b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280472704:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.912957:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x9e:0x0](ffff8ab3d1ef7448) 00000080:00200000:3.0:1787827203.912968:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub75, unhashed 1 00000080:00200000:3.0:1787827203.912982:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.912988:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.912994:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.913000:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.913005:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.913007:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.913009:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.913013:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.913015:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.913020:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.913026:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.913028:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.913030:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.913032:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.913034:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.913039:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub76, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.913041:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.913043:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.913046:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub76, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.913048:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub76, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.913063:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.913067:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af9680 x1874671280472960/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.913077:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af9680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280472960:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.913119:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.913980:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af9680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280472960:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.914004:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0x9f:0x0](ffff8ab3d1ee4b08) 00000080:00200000:3.0:1787827203.914015:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub76, unhashed 1 00000080:00200000:3.0:1787827203.914030:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.914036:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.914042:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.914048:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.914053:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.914056:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.914058:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.914061:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.914063:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.914069:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.914075:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.914077:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.914079:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.914081:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.914083:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.914088:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub77, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.914090:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.914092:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.914095:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub77, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.914097:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub77, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.914130:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.914135:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afbc00 x1874671280473216/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.914145:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afbc00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280473216:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.914171:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.915034:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afbc00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280473216:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.915048:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xa0:0x0](ffff8ab3d1ee63c8) 00000080:00200000:3.0:1787827203.915058:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub77, unhashed 1 00000080:00200000:3.0:1787827203.915072:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.915078:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.915084:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.915090:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.915095:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.915098:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.915113:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.915118:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.915120:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.915125:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.915132:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.915134:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.915137:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.915139:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.915141:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.915146:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub78, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.915148:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.915151:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.915153:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub78, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.915155:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub78, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.915183:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.915188:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0734780 x1874671280473472/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.915197:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280473472:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.915223:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.916035:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280473472:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.916050:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xa1:0x0](ffff8ab3d1ee2a08) 00000080:00200000:3.0:1787827203.916060:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub78, unhashed 1 00000080:00200000:3.0:1787827203.916075:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.916081:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.916087:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.916094:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.916099:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.916115:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.916117:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.916121:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.916124:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.916129:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.916135:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.916138:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.916141:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.916142:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.916144:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.916149:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub79, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.916151:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.916154:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.916156:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub79, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.916158:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub79, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.916175:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.916180:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0734f00 x1874671280473728/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.916189:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280473728:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.916215:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.917061:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280473728:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.917074:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xa2:0x0](ffff8ab3d1ee1988) 00000080:00200000:3.0:1787827203.917085:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub79, unhashed 1 00000080:00200000:3.0:1787827203.917113:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.917121:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.917126:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.917133:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.917138:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.917141:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.917142:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.917145:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.917148:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.917153:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.917159:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.917162:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.917164:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.917166:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.917167:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.917172:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub80, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.917174:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.917177:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.917179:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub80, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.917181:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub80, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.917197:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.917201:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f07361c0 x1874671280473984/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.917211:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07361c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280473984:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.917238:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.918087:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07361c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280473984:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.918113:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xa3:0x0](ffff8ab3d1ee1148) 00000080:00200000:3.0:1787827203.918125:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub80, unhashed 1 00000080:00200000:3.0:1787827203.918139:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.918145:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.918150:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.918157:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.918161:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.918164:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.918166:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.918169:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.918171:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.918176:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.918182:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.918184:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.918187:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.918189:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.918190:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.918195:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub81, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.918198:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.918200:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.918202:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub81, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.918204:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub81, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.918228:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.918233:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0736580 x1874671280474240/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.918242:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0736580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280474240:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.918268:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.919128:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0736580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280474240:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.919142:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xa4:0x0](ffff8ab3d1ee5b88) 00000080:00200000:3.0:1787827203.919164:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub81, unhashed 1 00000080:00200000:3.0:1787827203.919178:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.919185:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.919191:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.919198:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.919202:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.919205:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.919207:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.919210:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.919212:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.919218:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.919224:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.919226:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.919228:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.919230:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.919232:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.919237:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub82, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.919239:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.919242:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.919244:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub82, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.919246:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub82, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.919262:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.919266:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0737c00 x1874671280474496/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.919276:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0737c00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280474496:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.919301:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.920119:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0737c00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280474496:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.920138:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xa5:0x0](ffff8ab3d1ee0908) 00000080:00200000:3.0:1787827203.920153:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub82, unhashed 1 00000080:00200000:3.0:1787827203.920174:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.920182:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.920190:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.920198:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.920205:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.920210:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.920213:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.920216:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.920219:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.920226:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.920235:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.920239:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.920242:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.920245:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.920247:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.920255:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub83, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.920258:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.920262:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.920266:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub83, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.920269:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub83, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.920293:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.920299:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0736940 x1874671280474752/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.920314:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0736940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280474752:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.920341:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.921401:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0736940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280474752:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.921419:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xa6:0x0](ffff8ab3d1ee3248) 00000080:00200000:3.0:1787827203.921435:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub83, unhashed 1 00000080:00200000:3.0:1787827203.921458:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.921467:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.921474:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.921483:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.921488:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.921491:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.921493:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.921496:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.921499:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.921504:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.921511:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.921514:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.921517:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.921519:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.921521:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.921526:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub84, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.921529:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.921531:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.921534:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub84, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.921536:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub84, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.921553:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.921558:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0735a40 x1874671280475008/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.921569:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0735a40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280475008:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.921605:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.922543:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0735a40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280475008:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.922560:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xa7:0x0](ffff8ab3d1ee21c8) 00000080:00200000:3.0:1787827203.922571:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub84, unhashed 1 00000080:00200000:3.0:1787827203.922586:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.922593:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.922599:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.922606:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.922611:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.922613:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.922615:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.922619:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.922621:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.922626:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.922632:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.922635:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.922637:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.922639:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.922641:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.922646:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub85, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.922648:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.922651:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.922653:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub85, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.922655:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub85, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.922671:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.922676:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0734000 x1874671280475264/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.922685:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280475264:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.922712:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.923591:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280475264:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.923606:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xa8:0x0](ffff8ab3d1ee42c8) 00000080:00200000:3.0:1787827203.923617:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub85, unhashed 1 00000080:00200000:3.0:1787827203.923632:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.923638:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.923644:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.923651:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.923655:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.923658:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.923660:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.923663:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.923665:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.923671:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.923676:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.923679:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.923681:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.923683:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.923684:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.923689:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub86, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.923691:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.923694:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.923696:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub86, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.923698:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub86, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.923714:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.923719:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0735680 x1874671280475520/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.923728:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0735680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280475520:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.923754:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.924793:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0735680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280475520:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.924811:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xa9:0x0](ffff8ab3d1ee6c08) 00000080:00200000:3.0:1787827203.924825:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub86, unhashed 1 00000080:00200000:3.0:1787827203.924844:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.924853:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.924861:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.924871:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.924878:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.924882:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.924906:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.924911:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.924915:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.924925:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.924935:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.924940:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.924946:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.924950:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.924953:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.924961:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub87, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.924964:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.924968:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.924971:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub87, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.924974:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub87, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.924997:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.925003:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f07352c0 x1874671280475776/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.925018:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07352c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280475776:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.925052:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.926128:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07352c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280475776:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.926150:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xaa:0x0](ffff8ab3d1ee3a88) 00000080:00200000:3.0:1787827203.926165:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub87, unhashed 1 00000080:00200000:3.0:1787827203.926182:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.926189:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.926195:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.926202:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.926207:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.926210:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.926212:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.926216:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.926219:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.926224:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.926230:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.926233:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.926235:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.926237:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.926239:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.926245:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub88, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.926247:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.926250:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.926252:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub88, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.926254:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub88, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.926271:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.926277:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0737840 x1874671280476032/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.926287:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0737840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280476032:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.926315:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.927318:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0737840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280476032:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.927331:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xab:0x0](ffff8ab3d1ee5348) 00000080:00200000:3.0:1787827203.927342:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub88, unhashed 1 00000080:00200000:3.0:1787827203.927357:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.927364:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.927370:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.927376:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.927381:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.927384:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.927386:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.927389:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.927391:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.927397:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.927403:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.927405:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.927407:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.927409:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.927411:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.927417:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub89, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.927421:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.927425:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.927429:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub89, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.927432:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub89, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.927468:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.927475:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f07370c0 x1874671280476288/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.927486:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07370c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280476288:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.927518:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.928436:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07370c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280476288:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.928452:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xac:0x0](ffff8ab3d1ee7448) 00000080:00200000:3.0:1787827203.928463:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub89, unhashed 1 00000080:00200000:3.0:1787827203.928479:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.928485:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.928491:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.928499:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.928504:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.928507:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.928508:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.928511:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.928514:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.928519:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.928526:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.928528:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.928531:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.928533:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.928534:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.928540:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub90, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.928542:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.928545:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.928547:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub90, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.928549:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub90, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.928565:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.928570:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0737480 x1874671280476544/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.928580:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0737480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280476544:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.928606:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.929714:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0737480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280476544:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.929728:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xad:0x0](ffff8ab3d1ee00c8) 00000080:00200000:3.0:1787827203.929739:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub90, unhashed 1 00000080:00200000:3.0:1787827203.929754:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.929760:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.929766:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.929773:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.929777:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.929780:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.929782:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.929786:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.929788:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.929793:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.929800:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.929802:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.929804:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.929806:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.929808:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.929812:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub91, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.929815:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.929817:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.929820:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub91, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.929822:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub91, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.929837:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.929841:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f07343c0 x1874671280476800/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.929851:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07343c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280476800:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.929877:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.930808:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07343c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280476800:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.930835:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xae:0x0](ffff8ab3d1f963c8) 00000080:00200000:3.0:1787827203.930847:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub91, unhashed 1 00000080:00200000:3.0:1787827203.930862:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.930868:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.930874:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.930880:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.930902:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.930905:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.930907:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.930910:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.930912:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.930918:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.930924:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.930927:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.930929:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.930931:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.930932:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.930937:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub92, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.930939:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.930942:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.930944:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub92, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.930946:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub92, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.930962:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.930967:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0735e00 x1874671280477056/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.930976:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0735e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280477056:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.931002:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.932030:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0735e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280477056:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.932045:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xaf:0x0](ffff8ab3d1f92a08) 00000080:00200000:3.0:1787827203.932055:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub92, unhashed 1 00000080:00200000:3.0:1787827203.932070:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.932076:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.932081:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.932087:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.932092:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.932095:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.932097:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.932114:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.932117:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.932123:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.932129:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.932132:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.932135:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.932137:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.932138:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.932144:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub93, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.932146:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.932149:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.932151:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub93, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.932153:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub93, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.932169:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.932174:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0736d00 x1874671280477312/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.932183:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0736d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280477312:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.932209:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.933083:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0736d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280477312:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.933097:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xb0:0x0](ffff8ab3d1f91988) 00000080:00200000:3.0:1787827203.933122:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub93, unhashed 1 00000080:00200000:3.0:1787827203.933138:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.933144:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.933150:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.933156:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.933161:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.933164:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.933166:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.933169:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.933171:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.933177:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.933183:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.933185:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.933187:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.933189:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.933191:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.933196:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub94, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.933199:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.933201:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.933204:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub94, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.933206:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub94, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.933221:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.933226:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0734b40 x1874671280477568/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.933235:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280477568:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.933261:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.934236:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280477568:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.934250:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xb1:0x0](ffff8ab3d1f91148) 00000080:00200000:3.0:1787827203.934261:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub94, unhashed 1 00000080:00200000:3.0:1787827203.934275:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.934281:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.934287:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.934293:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.934298:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.934301:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.934303:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.934306:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.934308:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.934314:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.934319:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.934322:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.934324:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.934325:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.934327:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.934332:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub95, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.934334:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.934336:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.934339:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub95, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.934341:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub95, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.934357:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.934361:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e0ba70c0 x1874671280477824/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.934370:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0ba70c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280477824:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.934398:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.935275:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba70c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280477824:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.935289:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xb2:0x0](ffff8ab3d1f95b88) 00000080:00200000:3.0:1787827203.935299:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub95, unhashed 1 00000080:00200000:3.0:1787827203.935313:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.935319:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.935325:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.935332:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.935336:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.935339:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.935341:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.935344:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.935346:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.935351:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.935357:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.935359:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.935362:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.935364:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.935365:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.935370:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub96, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.935372:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.935375:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.935377:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub96, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.935379:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub96, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.935396:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.935402:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e0ba43c0 x1874671280478080/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.935417:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0ba43c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280478080:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.935450:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.936424:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba43c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280478080:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.936443:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xb3:0x0](ffff8ab3d1f90908) 00000080:00200000:3.0:1787827203.936457:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub96, unhashed 1 00000080:00200000:3.0:1787827203.936472:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.936478:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.936484:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.936490:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.936495:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.936498:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.936501:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.936506:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.936509:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.936517:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.936527:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.936531:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.936534:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.936538:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.936540:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.936547:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub97, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.936551:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.936555:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.936558:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub97, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.936562:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub97, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.936595:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.936600:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e0ba6940 x1874671280478336/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.936610:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0ba6940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280478336:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.936636:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.937679:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba6940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280478336:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.937695:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xb4:0x0](ffff8ab3d1f93248) 00000080:00200000:3.0:1787827203.937719:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub97, unhashed 1 00000080:00200000:3.0:1787827203.937738:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.937745:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.937752:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.937760:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.937766:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.937769:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.937771:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.937774:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.937777:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.937783:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.937789:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.937792:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.937794:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.937796:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.937798:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.937804:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub98, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.937806:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.937809:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.937811:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub98, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.937823:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub98, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.937851:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.937859:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e0ba4f00 x1874671280478592/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.937875:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0ba4f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280478592:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.937941:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.939005:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba4f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280478592:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.939020:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xb5:0x0](ffff8ab3d1f921c8) 00000080:00200000:3.0:1787827203.939032:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub98, unhashed 1 00000080:00200000:3.0:1787827203.939050:0:48432:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:3.0:1787827203.939057:0:48432:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827203.939063:0:48432:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.939070:0:48432:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.939075:0:48432:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827203.939078:0:48432:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:3.0:1787827203.939080:0:48432:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:3.0:1787827203.939083:0:48432:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e3bbd0 00010000:00010000:3.0:1787827203.939085:0:48432:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1787827203.939091:0:48432:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827203.939097:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.939116:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:3.0:1787827203.939121:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.939123:0:48432:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds1, flags=24 00000080:00200000:3.0:1787827203.939125:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 1 00000080:00200000:3.0:1787827203.939131:0:48432:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub99, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=1536 00000080:00200000:3.0:1787827203.939133:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.939136:0:48432:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), inode mode 41ed mask 3 00000080:00200000:3.0:1787827203.939138:0:48432:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub99, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8), flags=33060, excl=1 00000080:00200000:3.0:1787827203.939141:0:48432:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub99, dir=[0x200001b72:0x2:0x0](ffff8ab3f139e3c8) mode 100444 dev 0 00000020:00100000:3.0:1787827203.939160:0:48432:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1787827203.939165:0:48432:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e0ba7480 x1874671280478848/t0(0) o36->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827203.939175:0:48432:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0ba7480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280478848:192.168.203.124@tcp:36:createmany.0 00000100:00100000:3.0:1787827203.939203:0:48432:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3d2104cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827203.940134:0:48432:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba7480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48432:1874671280478848:192.168.203.124@tcp:36:createmany.0 00000080:00200000:3.0:1787827203.940149:0:48432:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b72:0xb6:0x0](ffff8ab3d1f942c8) 00000080:00200000:3.0:1787827203.940160:0:48432:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub99, unhashed 1 00000100:00100000:3.0:1787827203.974496:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827203.974747:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827203.974754:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc1b02 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x772721ad1f0d8c9e expref: -99 pid: 48441 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.974762:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d045d200 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:3.0:1787827203.974765:0:47650:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc1b02 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x772721ad1f0d8c9e expref: -99 pid: 48441 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.974772:0:47650:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc1b02 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x772721ad1f0d8c9e expref: -99 pid: 48441 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.974778:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc1b02 lrc: 3/0,0 mode: PR/PR res: [0x200001b72:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x772721ad1f0d8c9e expref: -99 pid: 48441 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.974792:0:47650:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0be00/0x268e1a602cc103c lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6ae0 expref: -99 pid: 48428 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827203.974862:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc1b02 lrc: 2/0,0 mode: --/PR res: [0x200001b72:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x772721ad1f0d8c9e expref: -99 pid: 48441 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.974868:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827203.974910:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc1b02 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x772721ad1f0d8c9e expref: -99 pid: 48441 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.974917:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d200/0x268e1a602cc1b02 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x772721ad1f0d8c9e expref: -99 pid: 48441 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827203.974927:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827203.974929:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827203.974933:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.003383:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.003599:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.003609:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1b2c lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x82:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d8d46 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.003621:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0760800 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.003628:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1b2c lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x82:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d8d46 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.003651:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.003663:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.003741:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1b2c lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x82:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d8d46 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.003748:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.003771:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1b2c lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x82:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d8d46 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.003778:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005aef010c/0x268e1a602cc1b2c lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x82:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d8d46 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.003792:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.003796:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.003804:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.007556:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.007565:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959e00/0x268e1a602cc1b3a lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x772721ad1f0d8d69 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.007590:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6959e00 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:3.0:1787827204.007597:0:47650:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959e00/0x268e1a602cc1b3a lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x772721ad1f0d8d69 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.007614:0:47650:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959e00/0x268e1a602cc1b3a lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x772721ad1f0d8d69 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.007629:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959e00/0x268e1a602cc1b3a lrc: 3/0,0 mode: PR/PR res: [0x240001b71:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x772721ad1f0d8d69 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.007649:0:47650:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7c00/0x268e1a602cc1b10 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d8ceb expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.007681:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959e00/0x268e1a602cc1b3a lrc: 2/0,0 mode: --/PR res: [0x240001b71:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d8d69 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.007690:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.007731:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959e00/0x268e1a602cc1b3a lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d8d69 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.007740:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959e00/0x268e1a602cc1b3a lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d8d69 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.007757:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.007762:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.007765:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.015765:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.015997:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.016007:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc1b5d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x82:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d8dcb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.016039:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c166e800 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.016048:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc1b5d lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x82:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d8dcb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.016077:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.016087:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.016136:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc1b5d lrc: 2/0,0 mode: --/PR res: [0x280000400:0x82:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d8dcb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.016143:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.016214:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc1b5d lrc: 1/0,0 mode: --/PR res: [0x280000400:0x82:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d8dcb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.016223:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc1b5d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x82:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d8dcb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.016246:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.016249:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.016251:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.028069:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.040818:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.050385:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.059622:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.068987:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.077358:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.077579:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.077590:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad77f645/0x268e1a602cc1c2f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x85:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d8fed expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.077604:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d09b7e00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.077610:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad77f645/0x268e1a602cc1c2f lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x85:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d8fed expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.077623:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.077631:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.077663:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad77f645/0x268e1a602cc1c2f lrc: 2/0,0 mode: --/PR res: [0x280000400:0x85:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d8fed expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.077669:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.077728:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad77f645/0x268e1a602cc1c2f lrc: 1/0,0 mode: --/PR res: [0x280000400:0x85:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d8fed expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.077737:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad77f645/0x268e1a602cc1c2f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x85:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d8fed expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.077752:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.077756:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.077760:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.084793:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787827204.084813:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f2896d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280490624:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.085805:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c52 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.085823:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c52 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9048 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.085838:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c52 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9048 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.085851:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c52 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9048 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.085867:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c52 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9048 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.085879:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.085913:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c52 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9048 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.085923:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c52 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9048 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.085934:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c52 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9048 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.085956:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c52 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d9048 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827204.085996:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f2896d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280490624:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.087214:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.087223:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c52 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x86:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9048 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.087235:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d09b7600 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.087241:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c52 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x86:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9048 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.087258:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.087268:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.087345:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c52 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x86:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9048 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.087354:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.087381:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c52 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x86:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9048 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.087391:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c52 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x86:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9048 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.087410:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.087413:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.087417:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.096253:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.096266:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c75 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x86:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d90a3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.096288:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d09b7600 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.096295:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c75 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x86:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d90a3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.096359:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0xb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.096371:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0xb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.096414:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c75 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x86:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d90a3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.096423:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.096452:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c75 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x86:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d90a3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.096462:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1c75 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x86:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d90a3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.096495:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.096499:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.096503:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.105282:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.105433:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.105440:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000363ace68/0x268e1a602cc1c98 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x87:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d90fe expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.105450:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0623c00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.105455:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000363ace68/0x268e1a602cc1c98 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x87:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d90fe expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.105468:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.105475:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.105500:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000363ace68/0x268e1a602cc1c98 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x87:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d90fe expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.105506:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.105565:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000363ace68/0x268e1a602cc1c98 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x87:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d90fe expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.105577:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000363ace68/0x268e1a602cc1c98 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x87:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d90fe expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.105595:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.105598:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.105600:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.114668:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.114863:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.114871:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000799c7467/0x268e1a602cc1cbb lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x87:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9159 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.114878:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0623000 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.114883:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000799c7467/0x268e1a602cc1cbb lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x87:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9159 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.114915:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0xd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.114922:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0xd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.114948:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000799c7467/0x268e1a602cc1cbb lrc: 2/0,0 mode: --/PR res: [0x280000400:0x87:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9159 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.114954:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.114970:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000799c7467/0x268e1a602cc1cbb lrc: 1/0,0 mode: --/PR res: [0x280000400:0x87:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9159 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.115013:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000799c7467/0x268e1a602cc1cbb lrc: 0/0,0 mode: --/PR res: [0x280000400:0x87:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9159 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.115029:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.115031:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.115034:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.125070:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.125265:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.125272:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e7a97dcb/0x268e1a602cc1cde lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x88:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d91b4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.125282:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0622400 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.125289:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e7a97dcb/0x268e1a602cc1cde lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x88:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d91b4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.125313:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0xe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.125323:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0xe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.125392:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e7a97dcb/0x268e1a602cc1cde lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x88:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d91b4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.125399:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.125416:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e7a97dcb/0x268e1a602cc1cde lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x88:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d91b4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.125422:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e7a97dcb/0x268e1a602cc1cde lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x88:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d91b4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.125432:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.125434:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.125438:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.134989:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.136232:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.145860:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.147112:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.156225:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.156582:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280498944:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827204.157384:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.165966:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.167346:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.176583:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280501248:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827204.177596:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.186311:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.186526:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.186534:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e78d8b72/0x268e1a602cc1db0 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x8b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d93d6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.186542:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c81b4600 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.186547:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e78d8b72/0x268e1a602cc1db0 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x8b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d93d6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.186560:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x14:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.186567:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x14:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.186591:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e78d8b72/0x268e1a602cc1db0 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x8b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d93d6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.186635:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.186655:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e78d8b72/0x268e1a602cc1db0 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x8b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d93d6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.186661:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e78d8b72/0x268e1a602cc1db0 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x8b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d93d6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.186672:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.186674:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.186677:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.187520:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294790784 00000100:00100000:3.0:1787827204.187525:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294790784 00000100:00100000:3.0:1787827204.187551:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 193 00000100:00100000:3.0:1787827204.187558:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d0646580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:76386:x1874671294790784:12345-192.168.203.124@tcp:105: 00010000:00010000:3.0:1787827204.187566:0:47645:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000067c86a9d/0x268e1a602cc1db7 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d93dd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.187574:0:47645:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000067c86a9d/0x268e1a602cc1db7 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d93dd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.187582:0:47645:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000067c86a9d/0x268e1a602cc1db7 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d93dd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.187599:0:47645:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000067c86a9d/0x268e1a602cc1db7 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d93dd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.187605:0:47645:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000067c86a9d/0x268e1a602cc1db7 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d93dd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.187611:0:47645:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000067c86a9d/0x268e1a602cc1db7 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x80000020000 nid: local remote: 0x772721ad1f0d93dd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.187625:0:47645:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8ab3c81b4400) 00000100:00100000:3.0:1787827204.187650:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d0646580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:76386:x1874671294790784:12345-192.168.203.124@tcp:105: Request processed in 92us (172us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827204.187656:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 193 00000100:00100000:3.0:1787827204.195828:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.197126:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294791040 00000100:00100000:3.0:1787827204.197134:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294791040 00000100:00100000:3.0:1787827204.197185:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 195 00000100:00100000:3.0:1787827204.197194:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d0646d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:75585:x1874671294791040:12345-192.168.203.124@tcp:105: 00010000:00010000:3.0:1787827204.197207:0:47645:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000019e1f18d/0x268e1a602cc1dda lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9438 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.197219:0:47645:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000019e1f18d/0x268e1a602cc1dda lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9438 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.197227:0:47645:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000019e1f18d/0x268e1a602cc1dda lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9438 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.197235:0:47645:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000019e1f18d/0x268e1a602cc1dda lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d9438 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.197242:0:47645:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000019e1f18d/0x268e1a602cc1dda lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d9438 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.197250:0:47645:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000019e1f18d/0x268e1a602cc1dda lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x80000020000 nid: local remote: 0x772721ad1f0d9438 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.197269:0:47645:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8ab3c81b4a00) 00000100:00100000:3.0:1787827204.197317:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d0646d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:75585:x1874671294791040:12345-192.168.203.124@tcp:105: Request processed in 121us (236us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827204.197342:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 195 00000100:00100000:3.0:1787827204.204852:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.205063:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.205075:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000024884ed7/0x268e1a602cc1df6 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x8c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d948c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.205095:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c81b5800 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.205163:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000024884ed7/0x268e1a602cc1df6 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x8c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d948c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.205185:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x16:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.205198:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x16:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.205244:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000024884ed7/0x268e1a602cc1df6 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x8c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d948c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.205253:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.205283:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000024884ed7/0x268e1a602cc1df6 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x8c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d948c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.205294:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000024884ed7/0x268e1a602cc1df6 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x8c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d948c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.205323:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.205329:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.205331:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.214011:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.214297:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280505856:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827204.215940:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280505856:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827204.222584:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787827204.222609:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0488b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280506752:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.223572:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000af5b3df7/0x268e1a602cc1e3c lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.223602:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000af5b3df7/0x268e1a602cc1e3c lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9542 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.223616:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000af5b3df7/0x268e1a602cc1e3c lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9542 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.223631:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000af5b3df7/0x268e1a602cc1e3c lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9542 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.223645:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000af5b3df7/0x268e1a602cc1e3c lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9542 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.223655:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.223665:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000af5b3df7/0x268e1a602cc1e3c lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9542 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.223672:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000af5b3df7/0x268e1a602cc1e3c lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9542 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.223678:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000af5b3df7/0x268e1a602cc1e3c lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9542 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.223684:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000af5b3df7/0x268e1a602cc1e3c lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d9542 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827204.223702:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0488b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280506752:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.225052:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.225062:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000af5b3df7/0x268e1a602cc1e3c lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x8d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9542 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.225074:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d08eaa00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.225082:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000af5b3df7/0x268e1a602cc1e3c lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x8d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9542 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.225113:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x18:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.225128:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x18:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.225184:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000af5b3df7/0x268e1a602cc1e3c lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x8d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9542 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.225274:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.225312:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000af5b3df7/0x268e1a602cc1e3c lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x8d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9542 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.225324:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000af5b3df7/0x268e1a602cc1e3c lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x8d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9542 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.225366:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.225369:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.225372:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.234274:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.241908:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787827204.241929:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f048b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280509056:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.242960:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005bb163e8/0x268e1a602cc1e82 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.242984:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005bb163e8/0x268e1a602cc1e82 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d95f8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.243000:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005bb163e8/0x268e1a602cc1e82 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d95f8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.243014:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005bb163e8/0x268e1a602cc1e82 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d95f8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.243030:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005bb163e8/0x268e1a602cc1e82 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d95f8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.243043:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.243059:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005bb163e8/0x268e1a602cc1e82 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d95f8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.243070:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005bb163e8/0x268e1a602cc1e82 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d95f8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.243080:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005bb163e8/0x268e1a602cc1e82 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d95f8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.243093:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005bb163e8/0x268e1a602cc1e82 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d95f8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827204.243124:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f048b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280509056:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.244412:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.244421:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005bb163e8/0x268e1a602cc1e82 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x8e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d95f8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.244431:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d08ea600 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.244437:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005bb163e8/0x268e1a602cc1e82 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x8e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d95f8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.244456:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x1a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.244469:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x1a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.244546:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005bb163e8/0x268e1a602cc1e82 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x8e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d95f8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.244554:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.244577:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005bb163e8/0x268e1a602cc1e82 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x8e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d95f8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.244583:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005bb163e8/0x268e1a602cc1e82 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x8e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d95f8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.244601:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.244604:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.244609:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.254690:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280510464:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827204.256813:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280510464:192.168.203.124@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1787827204.265797:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.265805:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000091f6f55f/0x268e1a602cc1ec8 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x8f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d96ae expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.265821:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d08ea400 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.265826:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000091f6f55f/0x268e1a602cc1ec8 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x8f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d96ae expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.265843:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x1c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.265853:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x1c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.265950:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000091f6f55f/0x268e1a602cc1ec8 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x8f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d96ae expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.265958:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.265984:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000091f6f55f/0x268e1a602cc1ec8 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x8f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d96ae expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.265991:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000091f6f55f/0x268e1a602cc1ec8 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x8f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d96ae expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.266016:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.266019:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.266021:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.273424:0:10956:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1787827204.273444:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0450f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280512512:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.274328:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000176cab70/0x268e1a602cc1eeb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.274349:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000176cab70/0x268e1a602cc1eeb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9709 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.274366:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000176cab70/0x268e1a602cc1eeb lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9709 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.274381:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000176cab70/0x268e1a602cc1eeb lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9709 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.274396:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000176cab70/0x268e1a602cc1eeb lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9709 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.274405:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.274416:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000176cab70/0x268e1a602cc1eeb lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9709 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.274428:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000176cab70/0x268e1a602cc1eeb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9709 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.274434:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000176cab70/0x268e1a602cc1eeb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9709 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.274441:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000176cab70/0x268e1a602cc1eeb lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d9709 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827204.274456:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0450f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280512512:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.275806:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.275815:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000176cab70/0x268e1a602cc1eeb lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9709 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.275824:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6d3a600 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.275829:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000176cab70/0x268e1a602cc1eeb lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x8f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9709 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.275848:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.275862:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.275972:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000176cab70/0x268e1a602cc1eeb lrc: 2/0,0 mode: --/PR res: [0x280000400:0x8f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9709 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.275984:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.276006:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000176cab70/0x268e1a602cc1eeb lrc: 1/0,0 mode: --/PR res: [0x280000400:0x8f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9709 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.276013:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000176cab70/0x268e1a602cc1eeb lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9709 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.276028:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.276030:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.276035:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.277035:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294793088 00000100:00100000:3.0:1787827204.277043:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294793088 00000100:00100000:3.0:1787827204.277090:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 211 00000100:00100000:3.0:1787827204.277119:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fc3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:78888:x1874671294793088:12345-192.168.203.124@tcp:105: 00010000:00010000:3.0:1787827204.277132:0:47645:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c5f94666/0x268e1a602cc1ef2 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9710 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.277145:0:47645:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c5f94666/0x268e1a602cc1ef2 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9710 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.277157:0:47645:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c5f94666/0x268e1a602cc1ef2 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9710 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.277168:0:47645:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c5f94666/0x268e1a602cc1ef2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d9710 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.277184:0:47645:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c5f94666/0x268e1a602cc1ef2 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d9710 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.277194:0:47645:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c5f94666/0x268e1a602cc1ef2 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x80000020000 nid: local remote: 0x772721ad1f0d9710 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.277220:0:47645:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8ab3c6d3a400) 00000100:00100000:3.0:1787827204.277269:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fc3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:78888:x1874671294793088:12345-192.168.203.124@tcp:105: Request processed in 166us (278us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827204.277305:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 211 00010000:00010000:3.0:1787827204.287167:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.287175:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000091d6f3f/0x268e1a602cc1f0e lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x90:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9764 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.287187:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6d3a200 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.287193:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000091d6f3f/0x268e1a602cc1f0e lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x90:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9764 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.287208:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x1e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.287217:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x1e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.287258:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000091d6f3f/0x268e1a602cc1f0e lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x90:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9764 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.287315:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.287339:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000091d6f3f/0x268e1a602cc1f0e lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x90:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9764 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.287345:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000091d6f3f/0x268e1a602cc1f0e lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x90:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9764 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.287358:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.287361:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.287367:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.288343:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294793344 00000100:00100000:3.0:1787827204.288350:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294793344 00000100:00100000:3.0:1787827204.288395:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 213 00000100:00100000:3.0:1787827204.288408:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22ffc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:75584:x1874671294793344:12345-192.168.203.124@tcp:105: 00010000:00010000:3.0:1787827204.288420:0:47645:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000008fa01072/0x268e1a602cc1f15 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d976b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.288429:0:47645:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000008fa01072/0x268e1a602cc1f15 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d976b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.288436:0:47645:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000008fa01072/0x268e1a602cc1f15 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d976b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.288444:0:47645:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000008fa01072/0x268e1a602cc1f15 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d976b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.288451:0:47645:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000008fa01072/0x268e1a602cc1f15 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d976b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.288463:0:47645:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000008fa01072/0x268e1a602cc1f15 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x80000020000 nid: local remote: 0x772721ad1f0d976b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.288486:0:47645:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8ab3c6d3aa00) 00000100:00100000:3.0:1787827204.288542:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22ffc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:75584:x1874671294793344:12345-192.168.203.124@tcp:105: Request processed in 133us (231us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827204.288565:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 213 00000100:00100000:3.0:1787827204.295927:0:10956:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1787827204.295948:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0452d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280514816:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.296923:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b92b747c/0x268e1a602cc1f31 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.296941:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b92b747c/0x268e1a602cc1f31 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d97bf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.296954:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b92b747c/0x268e1a602cc1f31 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d97bf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.296968:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b92b747c/0x268e1a602cc1f31 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d97bf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.296983:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b92b747c/0x268e1a602cc1f31 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d97bf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.297004:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.297017:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b92b747c/0x268e1a602cc1f31 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d97bf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.297024:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b92b747c/0x268e1a602cc1f31 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d97bf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.297030:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b92b747c/0x268e1a602cc1f31 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d97bf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.297038:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b92b747c/0x268e1a602cc1f31 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d97bf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827204.297052:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0452d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280514816:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.298456:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.298463:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b92b747c/0x268e1a602cc1f31 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x90:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d97bf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.298472:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6d3be00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.298477:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b92b747c/0x268e1a602cc1f31 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x90:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d97bf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.298493:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x1f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.298505:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x1f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.298591:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b92b747c/0x268e1a602cc1f31 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x90:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d97bf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.298601:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.298627:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b92b747c/0x268e1a602cc1f31 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x90:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d97bf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.298633:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b92b747c/0x268e1a602cc1f31 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x90:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d97bf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.298650:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.298657:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.298663:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.299643:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827204.299661:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294793600 00000100:00100000:3.0:1787827204.299667:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294793600 00000100:00100000:3.0:1787827204.299716:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 215 00000100:00100000:3.0:1787827204.299728:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d0981450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:75583:x1874671294793600:12345-192.168.203.124@tcp:105: 00010000:00010000:3.0:1787827204.299740:0:47645:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d59ca25b/0x268e1a602cc1f38 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d97c6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.299751:0:47645:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d59ca25b/0x268e1a602cc1f38 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d97c6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.299759:0:47645:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d59ca25b/0x268e1a602cc1f38 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d97c6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.299773:0:47645:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d59ca25b/0x268e1a602cc1f38 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d97c6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.299782:0:47645:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d59ca25b/0x268e1a602cc1f38 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d97c6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.299791:0:47645:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d59ca25b/0x268e1a602cc1f38 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x80000020000 nid: local remote: 0x772721ad1f0d97c6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.299815:0:47645:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8ab3c6d3ac00) 00000100:00100000:3.0:1787827204.299868:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d0981450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:75583:x1874671294793600:12345-192.168.203.124@tcp:105: Request processed in 140us (287us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827204.299914:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 215 00000100:00100000:3.0:1787827204.308565:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.308792:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.308800:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c6f7ebbd/0x268e1a602cc1f54 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x91:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d981a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.308811:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6d3b400 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.308816:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c6f7ebbd/0x268e1a602cc1f54 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x91:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d981a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.308832:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x20:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.308847:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x20:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.308952:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c6f7ebbd/0x268e1a602cc1f54 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x91:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d981a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.308958:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.308978:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c6f7ebbd/0x268e1a602cc1f54 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x91:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d981a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.308985:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c6f7ebbd/0x268e1a602cc1f54 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x91:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d981a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.308996:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.308998:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.309004:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.310090:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294793856 00000100:00100000:3.0:1787827204.310111:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294793856 00000100:00100000:3.0:1787827204.310147:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 217 00000100:00100000:3.0:1787827204.310154:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fd2c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:76386:x1874671294793856:12345-192.168.203.124@tcp:105: 00010000:00010000:3.0:1787827204.310162:0:47645:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd4dd75/0x268e1a602cc1f5b lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9821 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.310171:0:47645:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd4dd75/0x268e1a602cc1f5b lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9821 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.310179:0:47645:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd4dd75/0x268e1a602cc1f5b lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9821 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.310187:0:47645:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd4dd75/0x268e1a602cc1f5b lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d9821 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.310194:0:47645:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd4dd75/0x268e1a602cc1f5b lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d9821 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.310201:0:47645:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd4dd75/0x268e1a602cc1f5b lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x80000020000 nid: local remote: 0x772721ad1f0d9821 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.310217:0:47645:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8ab3c6d3b600) 00000100:00100000:3.0:1787827204.310279:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fd2c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:76386:x1874671294793856:12345-192.168.203.124@tcp:105: Request processed in 120us (217us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827204.310305:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 217 00000100:00100000:3.0:1787827204.318972:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.320156:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.326368:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787827204.326388:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0451680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280518272:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827204.329472:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.338260:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.346047:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.346358:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280520832:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827204.347148:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294795136 00000100:00100000:3.0:1787827204.347157:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294795136 00000100:00100000:3.0:1787827204.347195:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 225 00000100:00100000:3.0:1787827204.347203:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fcf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:75584:x1874671294795136:12345-192.168.203.124@tcp:105: 00010000:00010000:3.0:1787827204.347217:0:47645:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c6f7ebbd/0x268e1a602cc1fe7 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->12287] (req 0->12287) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d998d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.347237:0:47645:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c6f7ebbd/0x268e1a602cc1fe7 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->12287] (req 0->12287) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d998d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.347247:0:47645:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c6f7ebbd/0x268e1a602cc1fe7 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d998d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.347256:0:47645:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c6f7ebbd/0x268e1a602cc1fe7 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d998d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.347263:0:47645:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c6f7ebbd/0x268e1a602cc1fe7 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d998d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.347270:0:47645:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c6f7ebbd/0x268e1a602cc1fe7 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x80000020000 nid: local remote: 0x772721ad1f0d998d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.347289:0:47645:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8ab3c6d3b400) 00000100:00100000:3.0:1787827204.347330:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fcf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:75584:x1874671294795136:12345-192.168.203.124@tcp:105: Request processed in 123us (242us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827204.347363:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 225 00000100:00100000:3.0:1787827204.355051:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.356293:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.365085:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.365094:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc2026 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x94:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9a3c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.365123:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0896c00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.365129:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc2026 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x94:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9a3c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.365142:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x26:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.365149:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x26:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.365226:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc2026 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x94:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9a3c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.365234:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.365255:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc2026 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x94:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9a3c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.365262:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc2026 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x94:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9a3c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.365275:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.365278:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.365280:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.367121:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba5680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280523136:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827204.382569:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787827204.382588:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e38b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280525184:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.383449:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000048e8fe4/0x268e1a602cc206c lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.383470:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000048e8fe4/0x268e1a602cc206c lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9af2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.383481:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000048e8fe4/0x268e1a602cc206c lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9af2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.383491:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000048e8fe4/0x268e1a602cc206c lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9af2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.383507:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000048e8fe4/0x268e1a602cc206c lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9af2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.383521:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.383535:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000048e8fe4/0x268e1a602cc206c lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9af2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.383546:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000048e8fe4/0x268e1a602cc206c lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9af2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.383553:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000048e8fe4/0x268e1a602cc206c lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9af2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.383560:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000048e8fe4/0x268e1a602cc206c lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d9af2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827204.383577:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c6e38b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280525184:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.384799:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.384806:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000048e8fe4/0x268e1a602cc206c lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x95:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9af2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.384814:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6d3bc00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.384821:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000048e8fe4/0x268e1a602cc206c lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x95:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9af2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.384835:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x28:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.384845:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x28:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.384880:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000048e8fe4/0x268e1a602cc206c lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x95:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9af2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.384950:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.384972:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000048e8fe4/0x268e1a602cc206c lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x95:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9af2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.384980:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000048e8fe4/0x268e1a602cc206c lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x95:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9af2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.384993:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.384997:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.385000:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.385818:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827204.385830:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294796160 00000100:00100000:3.0:1787827204.385835:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294796160 00000100:00100000:3.0:1787827204.385870:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 233 00000100:00100000:3.0:1787827204.385878:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d0980450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77225:x1874671294796160:12345-192.168.203.124@tcp:105: 00010000:00010000:3.0:1787827204.385906:0:47645:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000091d6f3f/0x268e1a602cc2073 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9af9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.385915:0:47645:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000091d6f3f/0x268e1a602cc2073 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9af9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.385922:0:47645:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000091d6f3f/0x268e1a602cc2073 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9af9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.385930:0:47645:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000091d6f3f/0x268e1a602cc2073 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d9af9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.385936:0:47645:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000091d6f3f/0x268e1a602cc2073 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d9af9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.385943:0:47645:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000091d6f3f/0x268e1a602cc2073 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x80000020000 nid: local remote: 0x772721ad1f0d9af9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.385960:0:47645:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8ab3c6d3a200) 00000100:00100000:3.0:1787827204.385998:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d0980450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:77225:x1874671294796160:12345-192.168.203.124@tcp:105: Request processed in 119us (221us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827204.386010:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 233 00010000:00010000:3.0:1787827204.394183:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.394190:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc208f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x95:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9b4d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.394200:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f2858000 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.394204:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc208f lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x95:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9b4d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.394219:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x29:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.394228:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x29:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.394270:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc208f lrc: 2/0,0 mode: --/PR res: [0x280000400:0x95:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9b4d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.394279:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.394337:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc208f lrc: 1/0,0 mode: --/PR res: [0x280000400:0x95:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9b4d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.394344:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc208f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x95:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9b4d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.394360:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.394364:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.394366:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.395160:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294796416 00000100:00100000:3.0:1787827204.395166:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294796416 00000100:00100000:3.0:1787827204.395194:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 235 00000100:00100000:3.0:1787827204.395202:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fe1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:78888:x1874671294796416:12345-192.168.203.124@tcp:105: 00010000:00010000:3.0:1787827204.395210:0:47645:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e85bb314/0x268e1a602cc2096 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->114687] (req 0->114687) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9b54 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.395217:0:47645:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e85bb314/0x268e1a602cc2096 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->114687] (req 0->114687) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9b54 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.395224:0:47645:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e85bb314/0x268e1a602cc2096 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->114687) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9b54 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.395234:0:47645:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e85bb314/0x268e1a602cc2096 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->114687) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d9b54 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.395244:0:47645:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e85bb314/0x268e1a602cc2096 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->114687) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d9b54 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.395254:0:47645:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e85bb314/0x268e1a602cc2096 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->114687) gid 0 flags: 0x80000020000 nid: local remote: 0x772721ad1f0d9b54 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.395278:0:47645:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8ab3f2858600) 00000100:00100000:3.0:1787827204.395320:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fe1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:78888:x1874671294796416:12345-192.168.203.124@tcp:105: Request processed in 117us (190us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827204.395330:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 235 00000100:00100000:3.0:1787827204.395982:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280526592:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827204.402723:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787827204.402737:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d06443c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280527488:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.403518:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc20b2 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.403535:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc20b2 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9ba8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.403549:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc20b2 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9ba8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.403562:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc20b2 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9ba8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.403575:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc20b2 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9ba8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.403583:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.403592:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc20b2 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9ba8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.403599:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc20b2 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9ba8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.403604:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc20b2 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9ba8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.403609:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc20b2 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d9ba8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827204.403620:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d06443c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280527488:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.405016:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.405026:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc20b2 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x96:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9ba8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.405039:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0a07000 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.405048:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc20b2 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x96:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9ba8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.405065:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.405074:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.405174:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc20b2 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x96:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9ba8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.405185:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.405209:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc20b2 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x96:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9ba8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.405220:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc20b2 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x96:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9ba8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.405244:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.405251:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.405254:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.406122:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294796672 00000100:00100000:3.0:1787827204.406130:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294796672 00000100:00100000:3.0:1787827204.406168:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 237 00000100:00100000:3.0:1787827204.406175:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fde00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:75584:x1874671294796672:12345-192.168.203.124@tcp:105: 00010000:00010000:3.0:1787827204.406183:0:47645:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000609cf18d/0x268e1a602cc20b9 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->36863] (req 0->36863) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9baf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.406191:0:47645:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000609cf18d/0x268e1a602cc20b9 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->36863] (req 0->36863) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9baf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.406199:0:47645:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000609cf18d/0x268e1a602cc20b9 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->36863) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9baf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.406206:0:47645:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000609cf18d/0x268e1a602cc20b9 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->36863) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d9baf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.406212:0:47645:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000609cf18d/0x268e1a602cc20b9 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->36863) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d9baf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.406219:0:47645:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000609cf18d/0x268e1a602cc20b9 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->36863) gid 0 flags: 0x80000020000 nid: local remote: 0x772721ad1f0d9baf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.406237:0:47645:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8ab3d0a06e00) 00000100:00100000:3.0:1787827204.406268:0:47645:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fde00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:75584:x1874671294796672:12345-192.168.203.124@tcp:105: Request processed in 93us (197us total) trans 0 rc 0/0 00000100:00100000:3.0:1787827204.406276:0:47645:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 237 00000100:00100000:3.0:1787827204.406969:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280527744:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827204.413018:0:10956:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1787827204.413035:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d06443c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280528640:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.414004:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc20d5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.414021:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc20d5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9c03 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.414033:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc20d5 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9c03 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.414042:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc20d5 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9c03 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.414053:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc20d5 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9c03 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.414061:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.414071:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc20d5 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9c03 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.414077:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc20d5 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9c03 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.414083:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc20d5 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9c03 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.414091:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc20d5 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d9c03 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827204.414119:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d06443c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280528640:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.415466:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.415474:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc20d5 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x96:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9c03 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.415482:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0a07e00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.415487:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc20d5 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x96:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9c03 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.415502:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.415510:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.415577:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc20d5 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x96:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9c03 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.415584:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.415607:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc20d5 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x96:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9c03 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.415613:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc20d5 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x96:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9c03 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.415632:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.415634:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.415638:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.416780:0:47645:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1787827204.416797:0:47645:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294796928 00000100:00100000:3.0:1787827204.416803:0:47645:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294796928 00000100:00100000:3.0:1787827204.416850:0:47645:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 239 00000100:00100000:3.0:1787827204.416862:0:47645:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3f04bc850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:75583:x1874671294796928:12345-192.168.203.124@tcp:105: 00010000:00010000:3.0:1787827204.416872:0:47645:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009124e108/0x268e1a602cc20dc lrc: 5/0,1 mode: --/PW res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->16383] (req 0->16383) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9c0a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.416918:0:47645:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009124e108/0x268e1a602cc20dc lrc: 5/0,1 mode: --/PW res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->16383] (req 0->16383) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9c0a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.416932:0:47645:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009124e108/0x268e1a602cc20dc lrc: 5/0,1 mode: --/PW res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9c0a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.416945:0:47645:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009124e108/0x268e1a602cc20dc lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d9c0a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.416958:0:47645:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009124e108/0x268e1a602cc20dc lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x10080000020000 nid: local remote: 0x772721ad1f0d9c0a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.416968:0:47645:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009124e108/0x268e1a602cc20dc lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x80000020000 nid: local remote: 0x772721ad1f0d9c0a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787827204.417868:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280528896:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827204.425416:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.425615:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.425622:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ef38a859/0x268e1a602cc20f8 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x97:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9c5e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.425630:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0a06200 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.425635:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ef38a859/0x268e1a602cc20f8 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x97:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9c5e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.425647:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.425653:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.425684:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ef38a859/0x268e1a602cc20f8 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x97:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9c5e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.425691:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.425749:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ef38a859/0x268e1a602cc20f8 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x97:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9c5e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.425757:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ef38a859/0x268e1a602cc20f8 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x97:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9c5e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.425779:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.425785:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.425792:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.432981:0:10956:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1787827204.432997:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280530944:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.434083:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b5f86799/0x268e1a602cc211b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.434095:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b5f86799/0x268e1a602cc211b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9cb9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.434118:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b5f86799/0x268e1a602cc211b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9cb9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.434127:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b5f86799/0x268e1a602cc211b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9cb9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.434137:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b5f86799/0x268e1a602cc211b lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9cb9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.434144:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.434153:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b5f86799/0x268e1a602cc211b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9cb9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.434159:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b5f86799/0x268e1a602cc211b lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9cb9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.434164:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b5f86799/0x268e1a602cc211b lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9cb9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.434172:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b5f86799/0x268e1a602cc211b lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d9cb9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827204.434182:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280530944:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.435447:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.435455:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b5f86799/0x268e1a602cc211b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x97:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9cb9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.435465:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0a06400 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.435469:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b5f86799/0x268e1a602cc211b lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x97:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9cb9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.435481:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.435487:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.435552:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b5f86799/0x268e1a602cc211b lrc: 2/0,0 mode: --/PR res: [0x280000400:0x97:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9cb9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.435559:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.435580:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b5f86799/0x268e1a602cc211b lrc: 1/0,0 mode: --/PR res: [0x280000400:0x97:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9cb9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.435587:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b5f86799/0x268e1a602cc211b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x97:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9cb9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.435604:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.435607:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.435609:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.446327:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.446539:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.446547:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009094e625/0x268e1a602cc213e lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9d14 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.446556:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0a07a00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.446561:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009094e625/0x268e1a602cc213e lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9d14 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.446573:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.446579:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.446642:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009094e625/0x268e1a602cc213e lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9d14 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.446649:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.446671:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009094e625/0x268e1a602cc213e lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9d14 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.446681:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009094e625/0x268e1a602cc213e lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9d14 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.446697:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.446700:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.446703:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787827204.448436:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000008c240654 17ffffc0001001 3:0 ffff8ab3d06ffe80 8 lru 00000080:00200000:0.0:1787827204.448443:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000f08f7022 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 000000006a588f83 0000000096fb63da > 3< 0 0 > 4< 0 0 8 3715072 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.448446:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000091b56bdc 00000080:00200000:0.0:1787827204.448447:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 9. 00000080:00200000:0.0:1787827204.448449:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 9 of 0 to 978 bytes 00000080:00200000:0.0:1787827204.448469:0:48442:0:(rw26.c:875:ll_write_end()) pos 36864, len 978, copied 978 00000080:00200000:0.0:1787827204.448474:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000d31fbfc0[3 000000001a059290 2 1 000000002663f4b0 0 0 9] 00000080:00200000:0.0:1787827204.448478:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000afb45dae 17ffffc0001001 3:0 ffff8ab3d06ffae0 9 lru 00000080:00200000:0.0:1787827204.448484:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000005385f695 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 000000006a588f83 0000000096fb63da > 3< 0 0 > 4< 0 0 8 3715072 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.448487:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000d31fbfc0 00000080:00200000:0.0:1787827204.448488:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 10. 00000080:00200000:0.0:1787827204.448492:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 10, from 0, to 978 prio 0 00000008:00010000:0.0:1787827204.448521:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000c8153315 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.448542:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c8153315 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.448556:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c8153315 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.448567:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c8153315 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.448579:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c8153315 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.448592:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c8153315 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.448627:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c8153315 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.448639:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c8153315 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.448650:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c8153315 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.448662:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000c8153315 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.448681:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f51b38, count 10, dirtied 10 00000080:00200000:0.0:1787827204.448688:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 10 pages 37842 bytes, tot: 37842 00000080:00200000:0.0:1787827204.448691:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) replay-vbr.sh: write bytes 0, result: 37842 00000020:00010000:0.0:1787827204.448706:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.448710:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.448713:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f7b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.448719:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.448731:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.448744:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.448747:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f7b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.448755:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 37842 bytes 00000080:00200000:0.0:1787827204.448762:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 9] 00000080:00200000:0.0:1787827204.448768:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2e:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.448773:0:48442:0:(file.c:2240:ll_file_io_generic()) replay-vbr.sh: 2 io complete with rc: 0, result: 37842, restart: 0 00000080:00200000:0.0:1787827204.448779:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x2e:0x0] updating i_size 37842 i_blocks 0 00000080:00200000:0.0:1787827204.448784:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 37842 00000080:00200008:0.0:1787827204.448808:0:48442:0:(file.c:479:ll_file_release()) START file replay-vbr.sh:[0x240001b71:0x2e:0x0](ffff8ab3d1f51988), flags 100001 00000002:00100000:0.0:1787827204.448848:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3f0735e00 x1874671280531968/t17179869464(17179869464) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.448861:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.448868:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0737480 x1874671280532608/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.448922:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0737480 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280532608:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.448969:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c1377000 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.449850:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0737480 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280532608:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.449871:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3f0735e00 x1874671280531968/t17179869464(17179869464) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.449943:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.449957:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.449971:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.449982:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.449991:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.449997:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.450001:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.450006:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.450010:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.450019:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.450030:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.450036:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.450042:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.450045:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.450048:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.450052:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=replay-vbr.sh, flags=8 00000080:00200008:0.0:1787827204.450059:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file replay-vbr.sh:[0x240001b71:0x2e:0x0](ffff8ab3d1f51988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.450064:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x2e:0x0](ffff8ab3d1f51988),name=replay-vbr.sh 00000002:00010000:0.0:1787827204.450074:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x2e:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.450082:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07600/0x268e1a602cc2137 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9d06 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.450091:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07600/0x268e1a602cc2137 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9d06 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.450114:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.450119:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x2e:0x0](ffff8ab3d1f51988) 00000080:00010000:0.0:1787827204.450122:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x2e:0x0](ffff8ab3d1f51988) for lock 0x268e1a602cc2137 00000080:00010000:0.0:1787827204.450126:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2137 from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.450130:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07600/0x268e1a602cc2137 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9d06 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.450139:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07600/0x268e1a602cc2137 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9d06 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.450153:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.450160:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x2e:0x0] 00000008:00010000:0.0:1787827204.450169:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.450172:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f7b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.450184:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.450195:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.450206:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.450210:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.450219:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000abd45831/0x268e1a602cc2145 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d1b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.450228:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.450233:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x2e:0x0] updating i_size 37842 i_blocks 0 00000020:00010000:0.0:1787827204.450237:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.450240:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.450243:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f7b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.450245:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.450249:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f7b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.450255:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.450295:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.450304:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.450312:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.450321:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.450328:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.450332:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.450334:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.450338:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.450342:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.450350:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.450358:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.450363:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.450366:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.450369:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.450371:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.450383:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=rpc.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.450387:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=rpc.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.450396:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: rpc.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.450426:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc214c lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.450434:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc214c lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.450442:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc214c lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.450455:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07370c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280532736:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.450495:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c1377000 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.451422:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07370c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280532736:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.451431:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc214c lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.451438:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc214c lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.451443:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc214c lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.451447:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc214c lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.451452:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc214c lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.451456:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0a06a00 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.451460:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc214c lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.451468:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc214c lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.451473:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc214c lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.451477:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.451481:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc214c lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.451485:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc214c lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.451493:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3f07370c0 x1874671280532736/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 576/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.451501:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.451513:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.451537:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.451542:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.451547:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.451553:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.451558:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.451561:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.451563:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.451566:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.451568:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.451574:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.451580:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.451582:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.451585:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.451586:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.451588:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.451593:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=rpc.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.451595:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=rpc.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.451601:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: rpc.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.451615:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc2153 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.451620:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc2153 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.451625:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc2153 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.451634:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07370c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280532864:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.451663:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c1377000 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.452572:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07370c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280532864:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.452582:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc2153 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.452591:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc2153 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.452599:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc2153 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.452605:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc2153 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.452614:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc2153 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.452620:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0a06400 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.452624:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc2153 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.452631:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc2153 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.452638:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc2153 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.452644:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.452648:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc2153 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.452655:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc2153 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.452668:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3f07370c0 x1874671280532864/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 576/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.452678:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.452689:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.452743:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.452752:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.452758:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.452767:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.452774:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.452777:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.452780:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.452784:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.452788:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.452795:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.452804:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.452809:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.452812:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.452815:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.452817:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.452827:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.452832:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=rpc.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be2a80, open_flags 80c1, mode 81ed opened 0 00000080:00200000:0.0:1787827204.452839:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=rpc.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.452853:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: rpc.sh,[0x240001b71:0x2f:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.452881:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.452911:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.452922:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.452929:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f07370c0 x1874671280533120/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.452940:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.452955:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07370c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280533120:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.453042:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c1377000 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.454356:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07370c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280533120:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.454366:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.454378:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.454384:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.454391:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.454398:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.454408:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.454413:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.454419:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.454424:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3f07370c0 x1874671280533120/t17179869466(17179869466) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 648/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.454432:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3f07370c0 x1874671280533120/t17179869466(17179869466) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 648/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.454440:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.454447:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.454453:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/47 (0 0) 00000080:00200000:0.0:1787827204.454456:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.454462:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=rpc.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.454481:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x2f:0x0](ffff8ab3d1f51148) 00000080:00200000:0.0:1787827204.454540:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x2f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.454543:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x2f:0x0](ffff8ab3d1f51148) 00000080:00010000:0.0:1787827204.454546:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x2f:0x0](ffff8ab3d1f51148) for lock 0x268e1a602cc215a 00000080:00010000:0.0:1787827204.454553:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x2f:0x0](ffff8ab3d1f51148) 00000080:00200008:0.0:1787827204.454559:0:48442:0:(file.c:1090:ll_file_open()) START file rpc.sh:[0x240001b71:0x2f:0x0](ffff8ab3d1f51148), flags 100301 00000002:00100000:0.0:1787827204.454572:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3f07370c0 x1874671280533120/t17179869466(17179869466) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 576/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.454600:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc215a from it ffff8ab3d0a44f60 00010000:00010000:0.0:1787827204.454603:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.454618:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.454632:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x2f:0x0](ffff8ab3d1f51148), inode mode 81ed mask 40 00000080:00200008:0.0:1787827204.454642:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file rpc.sh:[0x240001b71:0x2f:0x0](ffff8ab3d1f51148), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.454645:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x2f:0x0](ffff8ab3d1f51148),name=rpc.sh 00000002:00010000:0.0:1787827204.454653:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x2f:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.454658:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.454665:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.454669:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.454672:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x2f:0x0](ffff8ab3d1f51148) 00000080:00010000:0.0:1787827204.454674:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x2f:0x0](ffff8ab3d1f51148) for lock 0x268e1a602cc215a 00000080:00010000:0.0:1787827204.454677:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc215a from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.454679:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.454689:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.454698:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2f:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.454702:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x2f:0x0] 00000008:00010000:0.0:1787827204.454709:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.454711:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7ffb0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.454731:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc2161 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.454738:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc2161 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.454760:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1787827204.455706:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x2f:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787827204.455713:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.455717:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.455719:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7ffb0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.455722:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.455725:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7ffb0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.455734:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2f:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.455759:0:48442:0:(file.c:2652:do_file_write_iter()) START file rpc.sh:[0x240001b71:0x2f:0x0], ppos: 0, count: 423 00000080:00200000:0.0:1787827204.455768:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 423 bytes 00000080:00200000:0.0:1787827204.455771:0:48442:0:(file.c:2628:ll_do_tiny_write()) result: 0, original count 423 00000080:00200000:0.0:1787827204.455774:0:48442:0:(file.c:2090:ll_file_io_generic()) rpc.sh: write ppos: 0, bytes: 423 00000080:00200000:0.0:1787827204.455778:0:48442:0:(file.c:1848:ll_io_set_mirror()) rpc.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.455782:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@0000000020fd8fd4[0x110, 1, [0x240001b71:0x2f:0x0] hash] 00000020:00200000:0.0:1787827204.455784:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 423) 0 0 00000080:00200000:0.0:1787827204.455787:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2f:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.455792:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1787827204.455798:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 423) 423, 423 00020000:00200000:0.0:1787827204.455801:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.455810:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1787827204.455818:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.455821:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7ffb0, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.455827:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3c7ca6800 type 11 mode 2 res 10737419264/152 (0 4095) 00010000:00010000:0.0:1787827204.455845:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.455852:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.455859:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.455870:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07352c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280533376:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.455919:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c1377d80 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.456572:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:0.0:1787827204.456618:0:48409:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294797824 00000100:00100000:0.0:1787827204.456624:0:48409:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294797824 00000100:00100000:0.0:1787827204.456665:0:48409:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 246 00000100:00100000:0.0:1787827204.456673:0:48409:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3e0ba43c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:77210:x1874671294797824:12345-192.168.203.124@tcp:104: 00010000:00010000:0.0:1787827204.456683:0:48409:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc2161 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9d6f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.456727:0:48409:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787827204.456737:0:48409:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3e0ba43c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:77210:x1874671294797824:12345-192.168.203.124@tcp:104: Request processed in 63us (164us total) trans 0 rc 0/0 00010000:00010000:3.0:1787827204.456748:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00000100:00100000:0.0:1787827204.456749:0:48409:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 246 00010000:00010000:3.0:1787827204.456756:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc2161 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9d6f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.456765:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0a07e00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.456769:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc2161 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9d6f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.456781:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.456787:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.456810:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc2161 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9d6f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.456816:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.456869:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc2161 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9d6f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.456878:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc2161 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9d6f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.456911:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.456914:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.456919:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.457284:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07352c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280533376:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.457310:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.457319:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.457326:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.457829:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.457846:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.457860:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.457870:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.457877:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.457907:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.457926:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) rpc.sh: write [0, 423) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.457934:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 423 bytes 00000080:00200000:0.0:1787827204.457969:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 423, copied 423 00000080:00200000:0.0:1787827204.457976:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000033c2be29[3 00000000f722dcad 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.457979:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000012e5d744 17ffffc0001001 3:0 ffff8ab3d06ffbc8 0 lru 00000080:00200000:0.0:1787827204.457985:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000006502fb51 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000f5856a27 000000001c9648e3 > 3< 0 0 > 4< 0 0 8 2973696 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.457987:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000033c2be29 00000080:00200000:0.0:1787827204.457988:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.457991:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 1, from 0, to 423 prio 0 00000008:00010000:0.0:1787827204.458004:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000005785d62c ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.458018:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f512f8, count 1, dirtied 1 00000080:00200000:0.0:1787827204.458023:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 1 pages 423 bytes, tot: 423 00000080:00200000:0.0:1787827204.458026:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) rpc.sh: write bytes 0, result: 423 00000020:00010000:0.0:1787827204.458039:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.458043:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.458046:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7ffb0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.458051:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.458061:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.458074:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.458078:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7ffb0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.458084:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 423 bytes 00000080:00200000:0.0:1787827204.458090:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1787827204.458108:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2f:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.458113:0:48442:0:(file.c:2240:ll_file_io_generic()) rpc.sh: 2 io complete with rc: 0, result: 423, restart: 0 00000080:00200000:0.0:1787827204.458119:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x2f:0x0] updating i_size 423 i_blocks 0 00000080:00200000:0.0:1787827204.458124:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 423 00000080:00200008:0.0:1787827204.458141:0:48442:0:(file.c:479:ll_file_release()) START file rpc.sh:[0x240001b71:0x2f:0x0](ffff8ab3d1f51148), flags 100001 00000002:00100000:0.0:1787827204.458179:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3f07370c0 x1874671280533120/t17179869466(17179869466) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 576/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.458191:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.458197:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f07352c0 x1874671280533760/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.458214:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07352c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280533760:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.458248:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c5feb3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.458596:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280533504:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787827204.459089:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07352c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280533760:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.459121:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3f07370c0 x1874671280533120/t17179869466(17179869466) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 576/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.459156:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.459165:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.459174:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.459184:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.459193:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.459198:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.459201:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.459207:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.459211:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.459221:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.459232:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.459238:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.459242:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.459244:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.459246:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.459256:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=rpc.sh, flags=8 00000080:00200008:0.0:1787827204.459261:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file rpc.sh:[0x240001b71:0x2f:0x0](ffff8ab3d1f51148), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.459264:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x2f:0x0](ffff8ab3d1f51148),name=rpc.sh 00000002:00010000:0.0:1787827204.459273:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x2f:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.459279:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.459286:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.459291:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.459293:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x2f:0x0](ffff8ab3d1f51148) 00000080:00010000:0.0:1787827204.459295:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x2f:0x0](ffff8ab3d1f51148) for lock 0x268e1a602cc215a 00000080:00010000:0.0:1787827204.459297:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc215a from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.459300:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.459310:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.459320:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2f:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.459325:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x2f:0x0] 00000008:00010000:0.0:1787827204.459331:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.459333:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7ffb0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.459339:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.459346:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.459353:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.459355:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.459361:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.459367:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.459370:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x2f:0x0] updating i_size 423 i_blocks 0 00000020:00010000:0.0:1787827204.459373:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.459375:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.459377:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7ffb0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.459379:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.459382:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7ffb0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.459386:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2f:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.459418:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.459427:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.459434:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.459444:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.459452:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.459456:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.459459:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.459462:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.459465:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.459471:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.459477:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.459479:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.459482:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.459484:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.459485:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.459501:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=run_dbench.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.459505:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_dbench.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.459512:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_dbench.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.459532:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07a00/0x268e1a602cc216f lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.459537:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07a00/0x268e1a602cc216f lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.459543:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07a00/0x268e1a602cc216f lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.459552:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0735680 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280533888:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.459583:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c5feb3c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.460410:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0735680 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280533888:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.460422:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07a00/0x268e1a602cc216f lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.460431:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07a00/0x268e1a602cc216f lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.460438:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07a00/0x268e1a602cc216f lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.460446:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07a00/0x268e1a602cc216f lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.460454:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07a00/0x268e1a602cc216f lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.460460:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0a07a00 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.460463:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07a00/0x268e1a602cc216f lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.460471:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07a00/0x268e1a602cc216f lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.460476:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07a00/0x268e1a602cc216f lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.460480:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.460483:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07a00/0x268e1a602cc216f lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.460487:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07a00/0x268e1a602cc216f lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.460496:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3f0735680 x1874671280533888/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.460504:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.460513:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.460534:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.460540:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.460544:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.460551:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.460556:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.460558:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.460560:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.460563:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.460565:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.460571:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.460577:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.460580:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.460582:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.460584:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.460585:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.460590:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=run_dbench.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.460593:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_dbench.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.460598:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_dbench.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.460614:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc2176 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.460619:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc2176 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.460625:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc2176 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.460633:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0735680 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280534016:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.460664:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c5feb3c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.461463:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0735680 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280534016:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.461470:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc2176 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.461480:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc2176 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.461484:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc2176 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.461489:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc2176 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.461493:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc2176 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.461496:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0a0a600 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.461498:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc2176 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.461502:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc2176 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.461506:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc2176 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.461509:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.461512:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc2176 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.461516:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc2176 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.461524:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3f0735680 x1874671280534016/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.461536:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.461544:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.461579:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.461584:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.461589:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.461595:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.461600:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.461603:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.461605:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.461608:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.461610:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.461615:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.461621:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.461624:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.461626:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.461628:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.461630:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.461636:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.461639:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=run_dbench.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be3b00, open_flags 80c1, mode 81ed opened 0 00000080:00200000:0.0:1787827204.461644:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_dbench.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.461652:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_dbench.sh,[0x240001b71:0x30:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.461672:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.461678:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.461683:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.461687:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0735680 x1874671280534272/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.461693:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.461702:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0735680 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280534272:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.461778:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c5feb3c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.462944:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0735680 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280534272:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.462954:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.462966:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.462973:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.462979:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.462991:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.462997:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.463002:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.463007:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.463014:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3f0735680 x1874671280534272/t17179869468(17179869468) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.463021:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3f0735680 x1874671280534272/t17179869468(17179869468) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.463030:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.463037:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.463043:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/48 (0 0) 00000080:00200000:0.0:1787827204.463046:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.463051:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=run_dbench.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.463066:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x30:0x0](ffff8ab3d1f55b88) 00000080:00200000:0.0:1787827204.463122:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x30:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.463126:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x30:0x0](ffff8ab3d1f55b88) 00000080:00010000:0.0:1787827204.463128:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x30:0x0](ffff8ab3d1f55b88) for lock 0x268e1a602cc217d 00000080:00010000:0.0:1787827204.463134:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x30:0x0](ffff8ab3d1f55b88) 00000080:00200008:0.0:1787827204.463140:0:48442:0:(file.c:1090:ll_file_open()) START file run_dbench.sh:[0x240001b71:0x30:0x0](ffff8ab3d1f55b88), flags 100301 00000002:00100000:0.0:1787827204.463152:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3f0735680 x1874671280534272/t17179869468(17179869468) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.463175:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc217d from it ffff8ab3d0a44f60 00010000:00010000:0.0:1787827204.463179:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.463186:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.463197:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x30:0x0](ffff8ab3d1f55b88), inode mode 81ed mask 40 00000080:00200008:0.0:1787827204.463205:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file run_dbench.sh:[0x240001b71:0x30:0x0](ffff8ab3d1f55b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.463209:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x30:0x0](ffff8ab3d1f55b88),name=run_dbench.sh 00000002:00010000:0.0:1787827204.463217:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x30:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.463223:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.463229:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.463234:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.463237:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x30:0x0](ffff8ab3d1f55b88) 00000080:00010000:0.0:1787827204.463239:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x30:0x0](ffff8ab3d1f55b88) for lock 0x268e1a602cc217d 00000080:00010000:0.0:1787827204.463243:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc217d from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.463245:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.463250:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.463258:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x30:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.463261:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x30:0x0] 00000008:00010000:0.0:1787827204.463267:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.463269:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7fa30, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.463288:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.463295:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.463316:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1787827204.463341:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787827204.463358:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280534400:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.464127:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.464139:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9dca expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.464147:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9dca expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.464155:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9dca expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.464162:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9dca expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.464169:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.464179:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9dca expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.464188:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9dca expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.464194:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9dca expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.464202:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d9dca expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.464208:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x30:0x0] updating i_size 0 i_blocks 0 00000100:00100000:3.0:1787827204.464213:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280534400:192.168.203.124@tcp:101:cp.0 00000020:00010000:0.0:1787827204.464214:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.464220:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.464222:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7fa30 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.464225:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.464227:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7fa30 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.464234:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x30:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.464251:0:48442:0:(file.c:2652:do_file_write_iter()) START file run_dbench.sh:[0x240001b71:0x30:0x0], ppos: 0, count: 1342 00000080:00200000:0.0:1787827204.464258:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 1342 bytes 00000080:00200000:0.0:1787827204.464260:0:48442:0:(file.c:2628:ll_do_tiny_write()) result: 0, original count 1342 00000080:00200000:0.0:1787827204.464263:0:48442:0:(file.c:2090:ll_file_io_generic()) run_dbench.sh: write ppos: 0, bytes: 1342 00000080:00200000:0.0:1787827204.464266:0:48442:0:(file.c:1848:ll_io_set_mirror()) run_dbench.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.464270:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@00000000ccc56b16[0x110, 1, [0x240001b71:0x30:0x0] hash] 00000020:00200000:0.0:1787827204.464271:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 1342) 0 0 00000080:00200000:0.0:1787827204.464273:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x30:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.464278:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1787827204.464282:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1342) 1342, 1342 00020000:00200000:0.0:1787827204.464284:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.464290:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1787827204.464296:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.464298:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7fa30, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.464304:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d2d3cc00 type 11 mode 2 res 11811161088/153 (0 4095) 00010000:00010000:0.0:1787827204.464321:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.464327:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.464334:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.464344:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0735a40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280534528:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.464375:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c1377d80 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.465202:0:48409:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294798080 00000100:00100000:0.0:1787827204.465207:0:48409:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294798080 00000100:00100000:0.0:1787827204.465236:0:48409:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 248 00000100:00100000:0.0:1787827204.465242:0:48409:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22ff0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:82330:x1874671294798080:12345-192.168.203.124@tcp:104: 00010000:00010000:0.0:1787827204.465250:0:48409:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9dca expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.465293:0:48409:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787827204.465298:0:48409:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22ff0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:82330:x1874671294798080:12345-192.168.203.124@tcp:104: Request processed in 57us (133us total) trans 0 rc 0/0 00000100:00100000:0.0:1787827204.465304:0:48409:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 248 00010000:00010000:3.0:1787827204.465316:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.465322:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9dca expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.465329:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c81b5600 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.465333:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9dca expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.465343:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x30:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.465349:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x30:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.465371:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9dca expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.465377:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.465394:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9dca expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.465400:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000059e16e96/0x268e1a602cc2184 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9dca expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.465470:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.465472:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.465475:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.465980:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0735a40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280534528:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.465991:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.465999:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.466005:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.466404:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.466416:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.466434:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.466442:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.466455:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.466462:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.466474:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) run_dbench.sh: write [0, 1342) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.466482:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 1342 bytes 00000080:00200000:0.0:1787827204.466505:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 1342, copied 1342 00000080:00200000:0.0:1787827204.466511:0:48442:0:(rw26.c:933:ll_write_end()) page@000000009bc84a2b[3 00000000dc4fff5d 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.466514:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000bad0bfa1 17ffffc0001001 3:0 ffff8ab3d06ff910 0 lru 00000080:00200000:0.0:1787827204.466519:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c84da8e8 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000006a588f83 000000006738e4be > 3< 0 0 > 4< 0 0 8 3649536 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.466521:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000009bc84a2b 00000080:00200000:0.0:1787827204.466522:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.466524:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 1, from 0, to 1342 prio 0 00000008:00010000:0.0:1787827204.466535:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000245f19da ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.466546:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f55d38, count 1, dirtied 1 00000080:00200000:0.0:1787827204.466549:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 1 pages 1342 bytes, tot: 1342 00000080:00200000:0.0:1787827204.466551:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) run_dbench.sh: write bytes 0, result: 1342 00000020:00010000:0.0:1787827204.466560:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.466562:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.466564:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7fa30 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.466568:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.466579:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.466587:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.466589:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7fa30 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.466593:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 1342 bytes 00000080:00200000:0.0:1787827204.466595:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1787827204.466599:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x30:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.466601:0:48442:0:(file.c:2240:ll_file_io_generic()) run_dbench.sh: 2 io complete with rc: 0, result: 1342, restart: 0 00000080:00200000:0.0:1787827204.466605:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x30:0x0] updating i_size 1342 i_blocks 0 00000080:00200000:0.0:1787827204.466608:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 1342 00000080:00200008:0.0:1787827204.466620:0:48442:0:(file.c:479:ll_file_release()) START file run_dbench.sh:[0x240001b71:0x30:0x0](ffff8ab3d1f55b88), flags 100001 00000002:00100000:0.0:1787827204.466641:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3f0735680 x1874671280534272/t17179869468(17179869468) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.466650:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.466653:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0735a40 x1874671280534912/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.466664:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0735a40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280534912:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.466699:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3e0c14f00 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.467461:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0735a40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280534912:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.467474:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3f0735680 x1874671280534272/t17179869468(17179869468) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.467502:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.467510:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.467518:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.467525:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.467531:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.467535:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.467537:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.467540:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.467543:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.467548:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.467555:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.467559:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.467562:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.467564:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.467566:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.467568:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=run_dbench.sh, flags=8 00000080:00200008:0.0:1787827204.467572:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file run_dbench.sh:[0x240001b71:0x30:0x0](ffff8ab3d1f55b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.467574:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x30:0x0](ffff8ab3d1f55b88),name=run_dbench.sh 00000002:00010000:0.0:1787827204.467580:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x30:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.467585:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.467595:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.467600:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.467602:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x30:0x0](ffff8ab3d1f55b88) 00000080:00010000:0.0:1787827204.467603:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x30:0x0](ffff8ab3d1f55b88) for lock 0x268e1a602cc217d 00000080:00010000:0.0:1787827204.467606:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc217d from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.467607:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.467612:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.467621:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x30:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.467625:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x30:0x0] 00000008:00010000:0.0:1787827204.467631:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.467633:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7fa30, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.467639:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.467646:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.467653:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.467655:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.467666:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.467672:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.467676:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x30:0x0] updating i_size 1342 i_blocks 0 00000020:00010000:0.0:1787827204.467679:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.467682:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.467685:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7fa30 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.467688:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.467691:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7fa30 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.467697:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x30:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.467730:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.467738:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.467744:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.467751:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.467756:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.467759:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.467761:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.467763:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.467766:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.467776:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.467782:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.467784:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.467786:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.467788:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.467790:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.467796:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=run_dd.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.467799:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_dd.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.467804:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_dd.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.467821:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5a00/0x268e1a602cc2192 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.467826:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5a00/0x268e1a602cc2192 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.467831:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5a00/0x268e1a602cc2192 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.467840:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0736940 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280535040:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.467871:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3e0c14f00 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.468786:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0736940 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280535040:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.468795:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5a00/0x268e1a602cc2192 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.468801:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5a00/0x268e1a602cc2192 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.468806:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5a00/0x268e1a602cc2192 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.468810:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5a00/0x268e1a602cc2192 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.468815:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5a00/0x268e1a602cc2192 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.468819:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c81b5a00 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.468821:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5a00/0x268e1a602cc2192 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.468825:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5a00/0x268e1a602cc2192 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.468829:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5a00/0x268e1a602cc2192 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.468833:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.468836:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5a00/0x268e1a602cc2192 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.468841:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5a00/0x268e1a602cc2192 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.468848:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3f0736940 x1874671280535040/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.468855:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.468864:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.468918:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.468929:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.468939:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.468949:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.468956:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.468960:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.468962:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.468965:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.468968:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.468973:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.468980:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.468983:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.468985:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.468987:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.468989:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.468994:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=run_dd.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.468996:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_dd.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.469002:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_dd.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.469017:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4600/0x268e1a602cc2199 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.469026:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4600/0x268e1a602cc2199 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.469031:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4600/0x268e1a602cc2199 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.469040:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0736940 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280535168:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.469067:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3e0c14f00 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.470011:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0736940 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280535168:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.470019:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4600/0x268e1a602cc2199 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.470025:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4600/0x268e1a602cc2199 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.470030:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4600/0x268e1a602cc2199 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.470034:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4600/0x268e1a602cc2199 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.470038:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4600/0x268e1a602cc2199 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.470042:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c81b4600 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.470044:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4600/0x268e1a602cc2199 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.470048:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4600/0x268e1a602cc2199 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.470053:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4600/0x268e1a602cc2199 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.470057:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.470060:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4600/0x268e1a602cc2199 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.470063:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4600/0x268e1a602cc2199 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.470071:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3f0736940 x1874671280535168/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.470077:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.470086:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.470132:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.470139:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.470144:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.470151:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.470156:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.470159:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.470161:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.470164:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.470166:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.470176:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.470183:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.470186:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.470188:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.470190:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.470191:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.470198:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.470200:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=run_dd.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be2a80, open_flags 80c1, mode 81ed opened 0 00000080:00200000:0.0:1787827204.470205:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_dd.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.470214:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_dd.sh,[0x240001b71:0x31:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.470233:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.470238:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.470244:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.470248:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0736940 x1874671280535424/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.470263:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.470273:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0736940 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280535424:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.470352:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3e0c14f00 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.471445:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0736940 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280535424:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.471454:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.471465:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.471470:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.471476:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.471482:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.471497:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.471502:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.471508:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.471514:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3f0736940 x1874671280535424/t17179869470(17179869470) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.471521:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3f0736940 x1874671280535424/t17179869470(17179869470) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.471530:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.471538:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.471544:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/49 (0 0) 00000080:00200000:0.0:1787827204.471547:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.471552:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=run_dd.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.471568:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x31:0x0](ffff8ab3d1f50908) 00000080:00200000:0.0:1787827204.471611:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x31:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.471614:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x31:0x0](ffff8ab3d1f50908) 00000080:00010000:0.0:1787827204.471617:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x31:0x0](ffff8ab3d1f50908) for lock 0x268e1a602cc21a0 00000080:00010000:0.0:1787827204.471622:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x31:0x0](ffff8ab3d1f50908) 00000080:00200008:0.0:1787827204.471628:0:48442:0:(file.c:1090:ll_file_open()) START file run_dd.sh:[0x240001b71:0x31:0x0](ffff8ab3d1f50908), flags 100301 00000002:00100000:0.0:1787827204.471639:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3f0736940 x1874671280535424/t17179869470(17179869470) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.471661:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc21a0 from it ffff8ab3d0a44f60 00010000:00010000:0.0:1787827204.471664:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.471671:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.471687:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x31:0x0](ffff8ab3d1f50908), inode mode 81ed mask 40 00000080:00200008:0.0:1787827204.471699:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file run_dd.sh:[0x240001b71:0x31:0x0](ffff8ab3d1f50908), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.471715:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x31:0x0](ffff8ab3d1f50908),name=run_dd.sh 00000002:00010000:0.0:1787827204.471727:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x31:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.471735:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.471745:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.471753:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.471758:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x31:0x0](ffff8ab3d1f50908) 00000080:00010000:0.0:1787827204.471761:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x31:0x0](ffff8ab3d1f50908) for lock 0x268e1a602cc21a0 00000080:00010000:0.0:1787827204.471766:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc21a0 from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.471769:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.471779:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.471793:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x31:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.471798:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x31:0x0] 00000008:00010000:0.0:1787827204.471805:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.471807:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f4b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.471848:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009094e625/0x268e1a602cc21a7 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.471856:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009094e625/0x268e1a602cc21a7 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.471879:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1787827204.472748:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x31:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787827204.472756:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.472764:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.472772:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f4b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.472777:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.472780:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f4b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.472793:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x31:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.472813:0:48442:0:(file.c:2652:do_file_write_iter()) START file run_dd.sh:[0x240001b71:0x31:0x0], ppos: 0, count: 1659 00000080:00200000:0.0:1787827204.472821:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 1659 bytes 00000080:00200000:0.0:1787827204.472824:0:48442:0:(file.c:2628:ll_do_tiny_write()) result: 0, original count 1659 00000080:00200000:0.0:1787827204.472826:0:48442:0:(file.c:2090:ll_file_io_generic()) run_dd.sh: write ppos: 0, bytes: 1659 00000080:00200000:0.0:1787827204.472830:0:48442:0:(file.c:1848:ll_io_set_mirror()) run_dd.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.472833:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@00000000ec56180d[0x110, 1, [0x240001b71:0x31:0x0] hash] 00000020:00200000:0.0:1787827204.472835:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 1659) 0 0 00000080:00200000:0.0:1787827204.472837:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x31:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.472842:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1787827204.472847:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1659) 1659, 1659 00020000:00200000:0.0:1787827204.472849:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.472857:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1787827204.472863:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.472865:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f4b0, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.472871:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3c7ca6800 type 11 mode 2 res 10737419264/153 (0 4095) 00010000:00010000:0.0:1787827204.472916:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 3/0,1 mode: --/PW res: [0x280000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.472928:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 3/0,1 mode: --/PW res: [0x280000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.472946:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 3/0,1 mode: --/PW res: [0x280000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.472960:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0736580 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280535680:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.472991:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c1377d80 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.473803:0:48409:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294798336 00000100:00100000:0.0:1787827204.473808:0:48409:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294798336 00000100:00100000:0.0:1787827204.473841:0:48409:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 250 00000100:00100000:0.0:1787827204.473848:0:48409:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fe580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:77209:x1874671294798336:12345-192.168.203.124@tcp:104: 00010000:00010000:0.0:1787827204.473855:0:48409:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009094e625/0x268e1a602cc21a7 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9e25 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.473950:0:48409:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787827204.473957:0:48409:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fe580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:77209:x1874671294798336:12345-192.168.203.124@tcp:104: Request processed in 110us (193us total) trans 0 rc 0/0 00000100:00100000:0.0:1787827204.473965:0:48409:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 250 00010000:00010000:3.0:1787827204.473966:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.473973:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009094e625/0x268e1a602cc21a7 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9e25 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.473982:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0a07a00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.473986:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009094e625/0x268e1a602cc21a7 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9e25 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.473997:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x31:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.474003:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x31:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.474058:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009094e625/0x268e1a602cc21a7 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9e25 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.474065:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.474082:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009094e625/0x268e1a602cc21a7 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9e25 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.474088:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009094e625/0x268e1a602cc21a7 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9e25 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.474110:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.474113:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.474116:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.474552:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0736580 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280535680:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.474564:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 4/0,1 mode: --/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.474581:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 4/0,1 mode: --/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.474588:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 4/0,1 mode: --/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.475114:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.475124:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.475144:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.475157:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.475169:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.475178:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.475198:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) run_dd.sh: write [0, 1659) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.475208:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 1659 bytes 00000080:00200000:0.0:1787827204.475236:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 1659, copied 1659 00000080:00200000:0.0:1787827204.475243:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000b06c6c83[3 000000006a7ef236 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.475245:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000a798217b 17ffffc0001001 3:0 ffff8ab3d06ff828 0 lru 00000080:00200000:0.0:1787827204.475251:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000005822f025 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000f5856a27 000000003057ad03 > 3< 0 0 > 4< 0 0 8 2945024 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.475253:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000b06c6c83 00000080:00200000:0.0:1787827204.475255:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.475257:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 1, from 0, to 1659 prio 0 00000008:00010000:0.0:1787827204.475270:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000fe948028 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.475284:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f50ab8, count 1, dirtied 1 00000080:00200000:0.0:1787827204.475288:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 1 pages 1659 bytes, tot: 1659 00000080:00200000:0.0:1787827204.475291:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) run_dd.sh: write bytes 0, result: 1659 00000020:00010000:0.0:1787827204.475302:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.475306:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.475310:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f4b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.475315:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.475325:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.475338:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.475340:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f4b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.475345:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 1659 bytes 00000080:00200000:0.0:1787827204.475348:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1787827204.475352:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x31:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.475355:0:48442:0:(file.c:2240:ll_file_io_generic()) run_dd.sh: 2 io complete with rc: 0, result: 1659, restart: 0 00000080:00200000:0.0:1787827204.475360:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x31:0x0] updating i_size 1659 i_blocks 0 00000080:00200000:0.0:1787827204.475362:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 1659 00000080:00200008:0.0:1787827204.475376:0:48442:0:(file.c:479:ll_file_release()) START file run_dd.sh:[0x240001b71:0x31:0x0](ffff8ab3d1f50908), flags 100001 00000002:00100000:0.0:1787827204.475397:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3f0736940 x1874671280535424/t17179869470(17179869470) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.475405:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.475409:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0736580 x1874671280536064/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.475419:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0736580 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280536064:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.475449:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c000 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.475792:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba6940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280535808:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787827204.476230:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0736580 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280536064:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.476243:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3f0736940 x1874671280535424/t17179869470(17179869470) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.476271:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.476279:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.476288:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.476298:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.476306:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.476312:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.476315:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.476321:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.476325:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.476334:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.476343:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.476349:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.476352:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.476354:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.476357:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.476359:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=run_dd.sh, flags=8 00000080:00200008:0.0:1787827204.476363:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file run_dd.sh:[0x240001b71:0x31:0x0](ffff8ab3d1f50908), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.476366:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x31:0x0](ffff8ab3d1f50908),name=run_dd.sh 00000002:00010000:0.0:1787827204.476374:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x31:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.476379:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.476386:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.476390:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.476393:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x31:0x0](ffff8ab3d1f50908) 00000080:00010000:0.0:1787827204.476395:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x31:0x0](ffff8ab3d1f50908) for lock 0x268e1a602cc21a0 00000080:00010000:0.0:1787827204.476397:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc21a0 from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.476399:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.476404:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.476414:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x31:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.476418:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x31:0x0] 00000008:00010000:0.0:1787827204.476423:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.476425:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f4b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.476432:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.476439:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.476446:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.476449:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.476455:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.476460:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.476463:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x31:0x0] updating i_size 1659 i_blocks 0 00000020:00010000:0.0:1787827204.476465:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.476467:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.476469:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f4b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.476471:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.476473:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f4b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.476478:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x31:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.476511:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.476517:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.476522:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.476528:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.476537:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.476540:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.476542:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.476544:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.476547:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.476552:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.476557:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.476559:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.476562:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.476564:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.476565:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.476572:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=run_IOR.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.476574:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_IOR.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.476580:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_IOR.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.476597:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc21b5 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.476602:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc21b5 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.476608:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc21b5 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.476617:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07361c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280536192:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.476648:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c000 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.477641:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07361c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280536192:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.477652:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc21b5 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.477660:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc21b5 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.477667:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc21b5 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.477674:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc21b5 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.477682:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc21b5 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.477687:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0a06a00 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.477691:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc21b5 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.477697:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc21b5 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.477703:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc21b5 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.477717:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.477721:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc21b5 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.477733:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06a00/0x268e1a602cc21b5 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.477745:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3f07361c0 x1874671280536192/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.477755:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.477767:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.477793:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.477801:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.477809:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.477825:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.477831:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.477835:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.477836:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.477840:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.477842:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.477848:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.477854:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.477857:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.477860:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.477862:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.477863:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.477870:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=run_IOR.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.477873:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_IOR.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.477878:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_IOR.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.477925:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc21bc lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.477935:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc21bc lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.477944:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc21bc lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.477956:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07361c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280536320:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.477986:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c000 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.478826:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07361c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280536320:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.478837:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc21bc lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.478846:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc21bc lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.478852:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc21bc lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.478858:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc21bc lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.478869:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc21bc lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.478875:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0a06400 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.478879:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc21bc lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.478909:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc21bc lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.478917:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc21bc lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.478923:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.478927:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc21bc lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.478933:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06400/0x268e1a602cc21bc lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.478945:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3f07361c0 x1874671280536320/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.478955:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.478966:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.479012:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.479019:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.479025:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.479036:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.479041:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.479044:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.479046:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.479049:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.479051:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.479056:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.479063:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.479066:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.479068:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.479070:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.479072:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.479079:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.479083:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=run_IOR.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be3080, open_flags 80c1, mode 81ed opened 0 00000080:00200000:0.0:1787827204.479088:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_IOR.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.479110:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_IOR.sh,[0x240001b71:0x32:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.479134:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.479139:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.479146:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.479152:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f07361c0 x1874671280536576/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.479161:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.479174:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07361c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280536576:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.479252:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c000 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.480474:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07361c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280536576:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.480484:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.480495:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.480500:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.480506:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.480512:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.480518:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.480527:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.480532:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.480537:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3f07361c0 x1874671280536576/t17179869472(17179869472) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.480544:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3f07361c0 x1874671280536576/t17179869472(17179869472) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.480552:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.480559:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.480565:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/50 (0 0) 00000080:00200000:0.0:1787827204.480568:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.480573:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=run_IOR.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.480591:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x32:0x0](ffff8ab3d1f53248) 00000080:00200000:0.0:1787827204.480634:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x32:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.480637:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x32:0x0](ffff8ab3d1f53248) 00000080:00010000:0.0:1787827204.480639:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x32:0x0](ffff8ab3d1f53248) for lock 0x268e1a602cc21c3 00000080:00010000:0.0:1787827204.480644:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x32:0x0](ffff8ab3d1f53248) 00000080:00200008:0.0:1787827204.480650:0:48442:0:(file.c:1090:ll_file_open()) START file run_IOR.sh:[0x240001b71:0x32:0x0](ffff8ab3d1f53248), flags 100301 00000002:00100000:0.0:1787827204.480661:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3f07361c0 x1874671280536576/t17179869472(17179869472) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.480685:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc21c3 from it ffff8ab3d0a44f60 00010000:00010000:0.0:1787827204.480689:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.480695:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.480706:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x32:0x0](ffff8ab3d1f53248), inode mode 81ed mask 40 00000080:00200008:0.0:1787827204.480714:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file run_IOR.sh:[0x240001b71:0x32:0x0](ffff8ab3d1f53248), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.480717:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x32:0x0](ffff8ab3d1f53248),name=run_IOR.sh 00000002:00010000:0.0:1787827204.480725:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x32:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.480732:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.480742:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.480749:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.480754:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x32:0x0](ffff8ab3d1f53248) 00000080:00010000:0.0:1787827204.480757:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x32:0x0](ffff8ab3d1f53248) for lock 0x268e1a602cc21c3 00000080:00010000:0.0:1787827204.480761:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc21c3 from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.480765:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.480774:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.480787:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x32:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.480791:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x32:0x0] 00000008:00010000:0.0:1787827204.480798:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.480800:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f830, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.480820:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.480827:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.480846:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1787827204.480867:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787827204.480880:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280536704:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.481677:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.481688:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9e80 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.481697:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9e80 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.481705:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9e80 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.481713:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9e80 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.481719:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.481728:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9e80 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.481734:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9e80 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.481744:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9e80 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.481751:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d9e80 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.481758:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x32:0x0] updating i_size 0 i_blocks 0 00000100:00100000:3.0:1787827204.481761:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280536704:192.168.203.124@tcp:101:cp.0 00000020:00010000:0.0:1787827204.481765:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.481771:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.481774:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f830 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.481778:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.481780:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f830 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.481787:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x32:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.481806:0:48442:0:(file.c:2652:do_file_write_iter()) START file run_IOR.sh:[0x240001b71:0x32:0x0], ppos: 0, count: 1753 00000080:00200000:0.0:1787827204.481812:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 1753 bytes 00000080:00200000:0.0:1787827204.481816:0:48442:0:(file.c:2628:ll_do_tiny_write()) result: 0, original count 1753 00000080:00200000:0.0:1787827204.481818:0:48442:0:(file.c:2090:ll_file_io_generic()) run_IOR.sh: write ppos: 0, bytes: 1753 00000080:00200000:0.0:1787827204.481821:0:48442:0:(file.c:1848:ll_io_set_mirror()) run_IOR.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.481825:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@00000000fbad4f08[0x110, 1, [0x240001b71:0x32:0x0] hash] 00000020:00200000:0.0:1787827204.481826:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 1753) 0 0 00000080:00200000:0.0:1787827204.481828:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x32:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.481834:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1787827204.481838:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1753) 1753, 1753 00020000:00200000:0.0:1787827204.481841:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.481849:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1787827204.481854:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.481856:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f830, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.481862:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d2d3cc00 type 11 mode 2 res 11811161088/154 (0 4095) 00010000:00010000:0.0:1787827204.481880:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.481906:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.481913:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.481925:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734780 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280536832:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.481957:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.482723:0:48409:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294798592 00000100:00100000:0.0:1787827204.482729:0:48409:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294798592 00000100:00100000:0.0:1787827204.482768:0:48409:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 252 00000100:00100000:0.0:1787827204.482779:0:48409:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22ff0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:77210:x1874671294798592:12345-192.168.203.124@tcp:104: 00010000:00010000:0.0:1787827204.482788:0:48409:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9e80 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.482828:0:48409:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787827204.482835:0:48409:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22ff0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:77210:x1874671294798592:12345-192.168.203.124@tcp:104: Request processed in 58us (162us total) trans 0 rc 0/0 00000100:00100000:0.0:1787827204.482841:0:48409:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 252 00010000:00010000:3.0:1787827204.482855:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.482861:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9e80 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.482869:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0896c00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.482874:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9e80 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.482906:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x32:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.482951:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x32:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.482972:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9e80 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.482978:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.482997:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9e80 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.483009:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e700f7/0x268e1a602cc21ca lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9e80 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.483019:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.483021:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.483024:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.483427:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734780 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280536832:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.483441:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.483449:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.483456:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.484091:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.484112:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.484132:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.484142:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.484151:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.484157:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.484176:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) run_IOR.sh: write [0, 1753) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.484184:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 1753 bytes 00000080:00200000:0.0:1787827204.484210:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 1753, copied 1753 00000080:00200000:0.0:1787827204.484216:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000b09fc139[3 000000005ceaf8e5 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.484219:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000066831aec 17ffffc0001001 3:0 ffff8ab3d06ffcb0 0 lru 00000080:00200000:0.0:1787827204.484224:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000061fd2c9e 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000006a588f83 00000000fa00ec91 > 3< 0 0 > 4< 0 0 8 3620864 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.484226:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000b09fc139 00000080:00200000:0.0:1787827204.484227:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.484230:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 1, from 0, to 1753 prio 0 00000008:00010000:0.0:1787827204.484241:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000368e44ef ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.484252:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f533f8, count 1, dirtied 1 00000080:00200000:0.0:1787827204.484255:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 1 pages 1753 bytes, tot: 1753 00000080:00200000:0.0:1787827204.484257:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) run_IOR.sh: write bytes 0, result: 1753 00000020:00010000:0.0:1787827204.484266:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.484269:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.484271:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f830 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.484275:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.484284:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.484296:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.484299:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f830 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.484305:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 1753 bytes 00000080:00200000:0.0:1787827204.484311:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1787827204.484316:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x32:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.484321:0:48442:0:(file.c:2240:ll_file_io_generic()) run_IOR.sh: 2 io complete with rc: 0, result: 1753, restart: 0 00000080:00200000:0.0:1787827204.484327:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x32:0x0] updating i_size 1753 i_blocks 0 00000080:00200000:0.0:1787827204.484331:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 1753 00000080:00200008:0.0:1787827204.484346:0:48442:0:(file.c:479:ll_file_release()) START file run_IOR.sh:[0x240001b71:0x32:0x0](ffff8ab3d1f53248), flags 100001 00000002:00100000:0.0:1787827204.484368:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3f07361c0 x1874671280536576/t17179869472(17179869472) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.484376:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.484379:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0734780 x1874671280537216/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.484389:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734780 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280537216:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.484420:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.485155:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734780 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280537216:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.485169:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3f07361c0 x1874671280536576/t17179869472(17179869472) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.485200:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.485208:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.485216:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.485223:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.485228:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.485231:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.485234:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.485237:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.485240:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.485245:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.485251:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.485255:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.485257:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.485259:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.485261:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.485263:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=run_IOR.sh, flags=8 00000080:00200008:0.0:1787827204.485267:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file run_IOR.sh:[0x240001b71:0x32:0x0](ffff8ab3d1f53248), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.485270:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x32:0x0](ffff8ab3d1f53248),name=run_IOR.sh 00000002:00010000:0.0:1787827204.485276:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x32:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.485283:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.485292:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.485300:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.485304:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x32:0x0](ffff8ab3d1f53248) 00000080:00010000:0.0:1787827204.485307:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x32:0x0](ffff8ab3d1f53248) for lock 0x268e1a602cc21c3 00000080:00010000:0.0:1787827204.485311:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc21c3 from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.485315:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.485331:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.485345:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x32:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.485349:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x32:0x0] 00000008:00010000:0.0:1787827204.485356:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.485357:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f830, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.485365:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.485372:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.485379:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.485382:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.485387:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000bc94222a/0x268e1a602cc21d1 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e87 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.485393:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.485396:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x32:0x0] updating i_size 1753 i_blocks 0 00000020:00010000:0.0:1787827204.485398:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.485400:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.485402:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f830 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.485404:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.485407:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f830 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.485411:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x32:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.485437:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.485444:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.485449:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.485455:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.485460:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.485463:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.485465:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.485467:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.485469:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.485474:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.485480:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.485483:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.485485:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.485487:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.485488:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.485505:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=run_iozone.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.485508:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_iozone.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.485514:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_iozone.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.485532:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2800/0x268e1a602cc21d8 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.485538:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2800/0x268e1a602cc21d8 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.485550:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2800/0x268e1a602cc21d8 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.485559:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280537344:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.485588:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.486482:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280537344:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.486491:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2800/0x268e1a602cc21d8 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.486497:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2800/0x268e1a602cc21d8 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.486502:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2800/0x268e1a602cc21d8 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.486506:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2800/0x268e1a602cc21d8 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.486511:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2800/0x268e1a602cc21d8 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.486514:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f05b2800 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.486517:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2800/0x268e1a602cc21d8 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.486525:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2800/0x268e1a602cc21d8 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.486530:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2800/0x268e1a602cc21d8 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.486533:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.486536:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2800/0x268e1a602cc21d8 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.486540:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2800/0x268e1a602cc21d8 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.486548:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3f0734f00 x1874671280537344/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.486556:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.486565:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.486585:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.486590:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.486595:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.486601:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.486606:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.486608:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.486610:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.486612:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.486615:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.486621:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.486627:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.486629:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.486632:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.486633:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.486635:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.486640:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=run_iozone.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.486642:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_iozone.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.486647:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_iozone.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.486661:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc21df lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.486666:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc21df lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.486671:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc21df lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.486679:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280537472:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.486715:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.487574:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280537472:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.487584:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc21df lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.487595:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc21df lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.487599:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc21df lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.487605:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc21df lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.487609:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc21df lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.487613:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f05b2400 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.487615:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc21df lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.487619:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc21df lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.487623:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc21df lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.487627:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.487630:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc21df lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.487635:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc21df lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.487643:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3f0734f00 x1874671280537472/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.487656:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.487666:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.487702:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.487711:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.487719:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.487730:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.487738:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.487743:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.487746:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.487751:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.487754:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.487761:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.487768:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.487772:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.487775:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.487777:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.487779:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.487787:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.487791:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=run_iozone.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be3b00, open_flags 80c1, mode 81ed opened 0 00000080:00200000:0.0:1787827204.487796:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_iozone.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.487807:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_iozone.sh,[0x240001b71:0x33:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.487828:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.487834:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.487839:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.487844:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0734f00 x1874671280537728/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.487850:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.487860:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280537728:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.487975:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.489208:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280537728:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.489218:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.489229:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.489235:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.489241:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.489252:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.489259:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.489263:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.489268:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.489274:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3f0734f00 x1874671280537728/t17179869474(17179869474) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.489284:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3f0734f00 x1874671280537728/t17179869474(17179869474) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.489296:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.489306:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.489316:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/51 (0 0) 00000080:00200000:0.0:1787827204.489321:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.489328:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=run_iozone.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.489353:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x33:0x0](ffff8ab3d1f521c8) 00000080:00200000:0.0:1787827204.489414:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x33:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.489418:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x33:0x0](ffff8ab3d1f521c8) 00000080:00010000:0.0:1787827204.489420:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x33:0x0](ffff8ab3d1f521c8) for lock 0x268e1a602cc21e6 00000080:00010000:0.0:1787827204.489427:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x33:0x0](ffff8ab3d1f521c8) 00000080:00200008:0.0:1787827204.489432:0:48442:0:(file.c:1090:ll_file_open()) START file run_iozone.sh:[0x240001b71:0x33:0x0](ffff8ab3d1f521c8), flags 100301 00000002:00100000:0.0:1787827204.489444:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3f0734f00 x1874671280537728/t17179869474(17179869474) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.489468:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc21e6 from it ffff8ab3d0a44f60 00010000:00010000:0.0:1787827204.489471:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.489478:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.489497:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x33:0x0](ffff8ab3d1f521c8), inode mode 81ed mask 40 00000080:00200008:0.0:1787827204.489508:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file run_iozone.sh:[0x240001b71:0x33:0x0](ffff8ab3d1f521c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.489511:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x33:0x0](ffff8ab3d1f521c8),name=run_iozone.sh 00000002:00010000:0.0:1787827204.489519:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x33:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.489524:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.489531:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.489535:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.489538:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x33:0x0](ffff8ab3d1f521c8) 00000080:00010000:0.0:1787827204.489539:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x33:0x0](ffff8ab3d1f521c8) for lock 0x268e1a602cc21e6 00000080:00010000:0.0:1787827204.489543:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc21e6 from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.489545:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.489550:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.489558:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x33:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.489562:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x33:0x0] 00000008:00010000:0.0:1787827204.489569:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.489570:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f9b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.489590:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ab80a841/0x268e1a602cc21ed lrc: 3/1,0 mode: --/PR res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.489596:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ab80a841/0x268e1a602cc21ed lrc: 3/1,0 mode: --/PR res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.489619:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1787827204.490556:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x33:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787827204.490561:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.490564:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.490568:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f9b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.490574:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.490577:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f9b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.490585:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x33:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.490605:0:48442:0:(file.c:2652:do_file_write_iter()) START file run_iozone.sh:[0x240001b71:0x33:0x0], ppos: 0, count: 1408 00000080:00200000:0.0:1787827204.490613:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 1408 bytes 00000080:00200000:0.0:1787827204.490616:0:48442:0:(file.c:2628:ll_do_tiny_write()) result: 0, original count 1408 00000080:00200000:0.0:1787827204.490619:0:48442:0:(file.c:2090:ll_file_io_generic()) run_iozone.sh: write ppos: 0, bytes: 1408 00000080:00200000:0.0:1787827204.490623:0:48442:0:(file.c:1848:ll_io_set_mirror()) run_iozone.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.490627:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@0000000085438647[0x110, 1, [0x240001b71:0x33:0x0] hash] 00000020:00200000:0.0:1787827204.490629:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 1408) 0 0 00000080:00200000:0.0:1787827204.490631:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x33:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.490637:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1787827204.490641:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1408) 1408, 1408 00020000:00200000:0.0:1787827204.490644:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.490652:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1787827204.490658:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.490661:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f9b0, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.490667:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3c7ca6800 type 11 mode 2 res 10737419264/154 (0 4095) 00010000:00010000:0.0:1787827204.490687:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.490697:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.490707:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.490723:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734000 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280537984:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.490762:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c540 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.491530:0:48409:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294798848 00000100:00100000:0.0:1787827204.491535:0:48409:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294798848 00000100:00100000:0.0:1787827204.491570:0:48409:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 254 00000100:00100000:0.0:1787827204.491576:0:48409:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fe580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:82330:x1874671294798848:12345-192.168.203.124@tcp:104: 00010000:00010000:0.0:1787827204.491585:0:48409:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ab80a841/0x268e1a602cc21ed lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9edb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.491629:0:48409:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787827204.491634:0:48409:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fe580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:82330:x1874671294798848:12345-192.168.203.124@tcp:104: Request processed in 58us (141us total) trans 0 rc 0/0 00000100:00100000:0.0:1787827204.491639:0:48409:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 254 00010000:00010000:3.0:1787827204.491658:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.491665:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ab80a841/0x268e1a602cc21ed lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9edb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.491673:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f05b3400 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.491678:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ab80a841/0x268e1a602cc21ed lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9edb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.491692:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x33:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.491700:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x33:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.491730:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ab80a841/0x268e1a602cc21ed lrc: 2/0,0 mode: --/PR res: [0x280000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9edb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.491738:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.491795:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ab80a841/0x268e1a602cc21ed lrc: 1/0,0 mode: --/PR res: [0x280000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9edb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.491804:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ab80a841/0x268e1a602cc21ed lrc: 0/0,0 mode: --/PR res: [0x280000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9edb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.491820:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.491822:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.491825:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.492335:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734000 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280537984:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.492346:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.492354:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.492360:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.492914:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.492926:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.492947:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.492957:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.492971:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.492977:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.492991:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) run_iozone.sh: write [0, 1408) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.492998:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 1408 bytes 00000080:00200000:0.0:1787827204.493027:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 1408, copied 1408 00000080:00200000:0.0:1787827204.493035:0:48442:0:(rw26.c:933:ll_write_end()) page@000000009d98bb0c[3 000000003b20985c 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.493038:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000006a516e92 17ffffc0001001 3:0 ffff8ab3d06ffd98 0 lru 00000080:00200000:0.0:1787827204.493043:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000d2e150e5 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000f5856a27 000000008426ec89 > 3< 0 0 > 4< 0 0 8 2916352 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.493045:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000009d98bb0c 00000080:00200000:0.0:1787827204.493046:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.493049:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 1, from 0, to 1408 prio 0 00000008:00010000:0.0:1787827204.493062:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000005ba2e828 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.493073:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f52378, count 1, dirtied 1 00000080:00200000:0.0:1787827204.493076:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 1 pages 1408 bytes, tot: 1408 00000080:00200000:0.0:1787827204.493078:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) run_iozone.sh: write bytes 0, result: 1408 00000020:00010000:0.0:1787827204.493088:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.493090:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.493093:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f9b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.493111:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.493123:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.493131:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.493133:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f9b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.493138:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 1408 bytes 00000080:00200000:0.0:1787827204.493142:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1787827204.493146:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x33:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.493149:0:48442:0:(file.c:2240:ll_file_io_generic()) run_iozone.sh: 2 io complete with rc: 0, result: 1408, restart: 0 00000080:00200000:0.0:1787827204.493153:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x33:0x0] updating i_size 1408 i_blocks 0 00000080:00200000:0.0:1787827204.493156:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 1408 00000080:00200008:0.0:1787827204.493171:0:48442:0:(file.c:479:ll_file_release()) START file run_iozone.sh:[0x240001b71:0x33:0x0](ffff8ab3d1f521c8), flags 100001 00000002:00100000:0.0:1787827204.493194:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3f0734f00 x1874671280537728/t17179869474(17179869474) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.493203:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.493207:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0734000 x1874671280538368/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.493217:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734000 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280538368:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.493247:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c540 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.493519:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280538112:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787827204.494075:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734000 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280538368:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.494089:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3f0734f00 x1874671280537728/t17179869474(17179869474) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.494138:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.494147:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.494156:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.494164:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.494169:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.494173:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.494175:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.494179:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.494181:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.494187:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.494194:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.494199:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.494202:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.494204:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.494206:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.494209:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=run_iozone.sh, flags=8 00000080:00200008:0.0:1787827204.494213:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file run_iozone.sh:[0x240001b71:0x33:0x0](ffff8ab3d1f521c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.494216:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x33:0x0](ffff8ab3d1f521c8),name=run_iozone.sh 00000002:00010000:0.0:1787827204.494222:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x33:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.494227:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.494237:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.494242:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.494244:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x33:0x0](ffff8ab3d1f521c8) 00000080:00010000:0.0:1787827204.494246:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x33:0x0](ffff8ab3d1f521c8) for lock 0x268e1a602cc21e6 00000080:00010000:0.0:1787827204.494248:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc21e6 from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.494249:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.494254:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.494263:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x33:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.494267:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x33:0x0] 00000008:00010000:0.0:1787827204.494274:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.494275:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f9b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.494285:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.494295:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.494305:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.494309:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.494326:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000317b5b4c/0x268e1a602cc21f4 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9ee2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.494336:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.494341:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x33:0x0] updating i_size 1408 i_blocks 0 00000020:00010000:0.0:1787827204.494344:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.494346:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.494348:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f9b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.494350:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.494352:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f9b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.494357:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x33:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.494386:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.494393:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.494398:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.494404:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.494409:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.494412:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.494414:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.494417:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.494419:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.494429:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.494435:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.494437:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.494439:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.494441:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.494443:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.494451:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=run-llog.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.494454:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run-llog.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.494460:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run-llog.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.494479:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b3a00/0x268e1a602cc21fb lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.494484:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b3a00/0x268e1a602cc21fb lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.494490:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b3a00/0x268e1a602cc21fb lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.494499:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0737840 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280538496:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.494529:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c540 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.495346:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0737840 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280538496:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.495354:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b3a00/0x268e1a602cc21fb lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.495361:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b3a00/0x268e1a602cc21fb lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.495366:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b3a00/0x268e1a602cc21fb lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.495370:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b3a00/0x268e1a602cc21fb lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.495375:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b3a00/0x268e1a602cc21fb lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.495378:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f05b3a00 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.495381:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b3a00/0x268e1a602cc21fb lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.495386:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b3a00/0x268e1a602cc21fb lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.495390:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b3a00/0x268e1a602cc21fb lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.495394:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.495398:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b3a00/0x268e1a602cc21fb lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.495402:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b3a00/0x268e1a602cc21fb lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.495411:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3f0737840 x1874671280538496/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.495419:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.495428:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.495449:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.495454:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.495459:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.495465:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.495470:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.495473:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.495475:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.495477:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.495479:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.495484:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.495491:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.495493:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.495496:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.495498:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.495499:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.495505:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=run-llog.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.495508:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run-llog.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.495513:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run-llog.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.495526:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2000/0x268e1a602cc2202 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.495535:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2000/0x268e1a602cc2202 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.495540:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2000/0x268e1a602cc2202 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.495549:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0737840 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280538624:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.495577:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c540 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.496368:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0737840 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280538624:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.496376:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2000/0x268e1a602cc2202 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.496382:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2000/0x268e1a602cc2202 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.496387:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2000/0x268e1a602cc2202 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.496391:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2000/0x268e1a602cc2202 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.496395:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2000/0x268e1a602cc2202 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.496399:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f05b2000 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.496401:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2000/0x268e1a602cc2202 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.496404:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2000/0x268e1a602cc2202 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.496409:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2000/0x268e1a602cc2202 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.496413:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.496416:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2000/0x268e1a602cc2202 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.496420:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2000/0x268e1a602cc2202 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.496427:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3f0737840 x1874671280538624/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.496433:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.496442:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.496477:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.496482:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.496487:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.496493:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.496498:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.496501:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.496503:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.496505:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.496508:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.496517:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.496523:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.496526:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.496528:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.496530:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.496532:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.496538:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.496541:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=run-llog.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be3080, open_flags 80c1, mode 81a4 opened 0 00000080:00200000:0.0:1787827204.496547:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run-llog.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.496557:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run-llog.sh,[0x240001b71:0x34:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.496576:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.496581:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.496587:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.496591:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0737840 x1874671280538880/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.496597:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.496606:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0737840 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280538880:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.496691:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c540 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.497864:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0737840 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280538880:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.497874:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.497916:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.497924:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.497931:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.497938:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.497945:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.497951:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.497956:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.497962:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3f0737840 x1874671280538880/t17179869476(17179869476) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.497970:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3f0737840 x1874671280538880/t17179869476(17179869476) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.497979:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.497986:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.497993:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/52 (0 0) 00000080:00200000:0.0:1787827204.497996:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.498002:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=run-llog.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.498019:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x34:0x0](ffff8ab3d1f542c8) 00000080:00200000:0.0:1787827204.498065:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x34:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.498068:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x34:0x0](ffff8ab3d1f542c8) 00000080:00010000:0.0:1787827204.498071:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x34:0x0](ffff8ab3d1f542c8) for lock 0x268e1a602cc2209 00000080:00010000:0.0:1787827204.498076:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x34:0x0](ffff8ab3d1f542c8) 00000080:00200008:0.0:1787827204.498082:0:48442:0:(file.c:1090:ll_file_open()) START file run-llog.sh:[0x240001b71:0x34:0x0](ffff8ab3d1f542c8), flags 100301 00000002:00100000:0.0:1787827204.498094:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3f0737840 x1874671280538880/t17179869476(17179869476) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.498136:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2209 from it ffff8ab3d0a44f60 00010000:00010000:0.0:1787827204.498140:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.498146:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.498157:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x34:0x0](ffff8ab3d1f542c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1787827204.498166:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file run-llog.sh:[0x240001b71:0x34:0x0](ffff8ab3d1f542c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.498170:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x34:0x0](ffff8ab3d1f542c8),name=run-llog.sh 00000002:00010000:0.0:1787827204.498178:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x34:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.498184:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.498190:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.498195:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.498198:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x34:0x0](ffff8ab3d1f542c8) 00000080:00010000:0.0:1787827204.498200:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x34:0x0](ffff8ab3d1f542c8) for lock 0x268e1a602cc2209 00000080:00010000:0.0:1787827204.498203:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2209 from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.498205:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.498210:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.498218:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x34:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.498222:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x34:0x0] 00000008:00010000:0.0:1787827204.498229:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.498230:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f2b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.498258:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000004b1a7c23/0x268e1a602cc2210 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.498265:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000004b1a7c23/0x268e1a602cc2210 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.498288:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1787827204.499206:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x34:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787827204.499214:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.499220:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.499224:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f2b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.499229:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.499234:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f2b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.499243:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x34:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.499261:0:48442:0:(file.c:2652:do_file_write_iter()) START file run-llog.sh:[0x240001b71:0x34:0x0], ppos: 0, count: 855 00000080:00200000:0.0:1787827204.499268:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 855 bytes 00000080:00200000:0.0:1787827204.499270:0:48442:0:(file.c:2628:ll_do_tiny_write()) result: 0, original count 855 00000080:00200000:0.0:1787827204.499273:0:48442:0:(file.c:2090:ll_file_io_generic()) run-llog.sh: write ppos: 0, bytes: 855 00000080:00200000:0.0:1787827204.499277:0:48442:0:(file.c:1848:ll_io_set_mirror()) run-llog.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.499281:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@000000007334cdd5[0x110, 1, [0x240001b71:0x34:0x0] hash] 00000020:00200000:0.0:1787827204.499282:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 855) 0 0 00000080:00200000:0.0:1787827204.499284:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x34:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.499290:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1787827204.499293:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 855) 855, 855 00020000:00200000:0.0:1787827204.499296:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.499303:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1787827204.499309:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.499312:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f2b0, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.499317:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d2d3cc00 type 11 mode 2 res 11811161088/155 (0 4095) 00010000:00010000:0.0:1787827204.499334:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.499340:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.499351:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.499361:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734b40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280539136:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.499392:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c480 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.500075:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:0.0:1787827204.500117:0:48409:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294799104 00000100:00100000:0.0:1787827204.500122:0:48409:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294799104 00000100:00100000:0.0:1787827204.500154:0:48409:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 256 00000100:00100000:0.0:1787827204.500159:0:48409:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3e0ba7480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:77209:x1874671294799104:12345-192.168.203.124@tcp:104: 00010000:00010000:0.0:1787827204.500167:0:48409:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000004b1a7c23/0x268e1a602cc2210 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9f36 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.500217:0:48409:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787827204.500225:0:48409:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3e0ba7480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:77209:x1874671294799104:12345-192.168.203.124@tcp:104: Request processed in 65us (151us total) trans 0 rc 0/0 00000100:00100000:0.0:1787827204.500235:0:48409:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 256 00010000:00010000:3.0:1787827204.500245:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.500252:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000004b1a7c23/0x268e1a602cc2210 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9f36 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.500262:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f05b2800 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.500267:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000004b1a7c23/0x268e1a602cc2210 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9f36 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.500278:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x34:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.500285:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x34:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.500306:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000004b1a7c23/0x268e1a602cc2210 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9f36 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.500312:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.500365:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000004b1a7c23/0x268e1a602cc2210 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9f36 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.500372:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000004b1a7c23/0x268e1a602cc2210 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9f36 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.500387:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.500390:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.500392:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.500926:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734b40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280539136:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.500938:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.500945:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.500952:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.501363:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.501383:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.501393:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.501401:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.501410:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.501420:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.501439:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) run-llog.sh: write [0, 855) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.501448:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 855 bytes 00000080:00200000:0.0:1787827204.501477:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 855, copied 855 00000080:00200000:0.0:1787827204.501486:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000eb858eea[3 000000001a68c983 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.501490:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000759f28de 17ffffc0001001 3:0 ffff8ab3d06ff2b8 0 lru 00000080:00200000:0.0:1787827204.501498:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000006f37578a 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000006a588f83 000000001ac9a404 > 3< 0 0 > 4< 0 0 8 3592192 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.501501:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000eb858eea 00000080:00200000:0.0:1787827204.501503:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.501507:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 1, from 0, to 855 prio 0 00000008:00010000:0.0:1787827204.501519:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 0000000085db1b04 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.501532:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f54478, count 1, dirtied 1 00000080:00200000:0.0:1787827204.501535:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 1 pages 855 bytes, tot: 855 00000080:00200000:0.0:1787827204.501538:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) run-llog.sh: write bytes 0, result: 855 00000020:00010000:0.0:1787827204.501547:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.501549:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.501551:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f2b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.501555:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.501561:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.501569:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.501570:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f2b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.501574:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 855 bytes 00000080:00200000:0.0:1787827204.501577:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1787827204.501581:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x34:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.501584:0:48442:0:(file.c:2240:ll_file_io_generic()) run-llog.sh: 2 io complete with rc: 0, result: 855, restart: 0 00000080:00200000:0.0:1787827204.501588:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x34:0x0] updating i_size 855 i_blocks 0 00000080:00200000:0.0:1787827204.501590:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 855 00000080:00200008:0.0:1787827204.501603:0:48442:0:(file.c:479:ll_file_release()) START file run-llog.sh:[0x240001b71:0x34:0x0](ffff8ab3d1f542c8), flags 100001 00000002:00100000:0.0:1787827204.501627:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3f0737840 x1874671280538880/t17179869476(17179869476) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.501635:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.501639:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0734b40 x1874671280539520/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.501649:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0734b40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280539520:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.501679:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c480 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.502430:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0734b40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280539520:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.502443:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3f0737840 x1874671280538880/t17179869476(17179869476) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.502472:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.502479:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.502486:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.502493:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.502499:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.502502:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.502504:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.502507:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.502509:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.502516:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.502522:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.502527:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.502530:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.502532:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.502534:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.502536:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=run-llog.sh, flags=8 00000080:00200008:0.0:1787827204.502539:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file run-llog.sh:[0x240001b71:0x34:0x0](ffff8ab3d1f542c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.502542:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x34:0x0](ffff8ab3d1f542c8),name=run-llog.sh 00000002:00010000:0.0:1787827204.502548:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x34:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.502552:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.502558:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.502563:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.502565:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x34:0x0](ffff8ab3d1f542c8) 00000080:00010000:0.0:1787827204.502567:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x34:0x0](ffff8ab3d1f542c8) for lock 0x268e1a602cc2209 00000080:00010000:0.0:1787827204.502569:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2209 from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.502571:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.502576:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.502585:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x34:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.502588:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x34:0x0] 00000008:00010000:0.0:1787827204.502594:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.502595:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f2b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.502601:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.502609:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.502616:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.502618:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.502624:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050dc31c9/0x268e1a602cc2217 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9f3d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.502630:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.502633:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x34:0x0] updating i_size 855 i_blocks 0 00000020:00010000:0.0:1787827204.502635:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.502637:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.502639:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f2b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.502641:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.502642:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f2b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.502647:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x34:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.502668:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.502674:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.502678:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.502684:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.502693:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.502696:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.502698:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.502700:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.502703:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.502708:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.502713:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.502715:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.502717:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.502719:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.502720:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.502727:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=run_tar.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.502731:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_tar.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.502740:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_tar.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.502763:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc221e lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.502771:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc221e lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.502781:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc221e lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.502794:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07343c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280539648:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.502824:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c480 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.503737:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07343c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280539648:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.503748:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc221e lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.503757:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc221e lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.503764:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc221e lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.503771:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc221e lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.503779:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc221e lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.503786:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0760800 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.503790:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc221e lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.503797:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc221e lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.503805:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc221e lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.503811:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.503815:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc221e lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.503828:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc221e lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.503841:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3f07343c0 x1874671280539648/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.503851:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.503861:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.503922:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.503931:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.503938:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.503948:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.503956:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.503960:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.503963:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.503967:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.503970:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.503979:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.503989:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.503993:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.503998:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.504001:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.504004:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.504012:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=run_tar.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.504015:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_tar.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.504024:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_tar.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.504045:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958000/0x268e1a602cc2225 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.504053:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958000/0x268e1a602cc2225 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.504061:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958000/0x268e1a602cc2225 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.504074:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07343c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280539776:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.504125:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c480 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.505041:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07343c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280539776:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.505049:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958000/0x268e1a602cc2225 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.505055:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958000/0x268e1a602cc2225 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.505059:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958000/0x268e1a602cc2225 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.505064:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958000/0x268e1a602cc2225 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.505072:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958000/0x268e1a602cc2225 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.505076:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6958000 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.505079:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958000/0x268e1a602cc2225 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.505082:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958000/0x268e1a602cc2225 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.505086:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958000/0x268e1a602cc2225 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.505089:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.505092:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958000/0x268e1a602cc2225 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.505108:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958000/0x268e1a602cc2225 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.505117:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3f07343c0 x1874671280539776/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.505124:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.505133:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.505166:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.505171:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.505176:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.505187:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.505192:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.505195:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.505197:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.505200:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.505202:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.505208:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.505214:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.505217:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.505220:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.505222:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.505223:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.505229:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.505232:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=run_tar.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be3b00, open_flags 80c1, mode 81ed opened 0 00000080:00200000:0.0:1787827204.505237:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=run_tar.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.505245:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: run_tar.sh,[0x240001b71:0x35:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.505272:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.505278:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.505284:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.505289:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f07343c0 x1874671280540032/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.505295:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.505304:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07343c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280540032:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.505366:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c480 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.506492:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07343c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280540032:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.506504:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.506520:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.506529:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.506539:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.506549:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.506561:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.506572:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.506577:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.506583:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3f07343c0 x1874671280540032/t17179869478(17179869478) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.506591:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3f07343c0 x1874671280540032/t17179869478(17179869478) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.506599:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.506607:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.506613:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/53 (0 0) 00000080:00200000:0.0:1787827204.506616:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.506622:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=run_tar.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.506636:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x35:0x0](ffff8ab3d1f56c08) 00000080:00200000:0.0:1787827204.506677:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x35:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.506681:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x35:0x0](ffff8ab3d1f56c08) 00000080:00010000:0.0:1787827204.506683:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x35:0x0](ffff8ab3d1f56c08) for lock 0x268e1a602cc222c 00000080:00010000:0.0:1787827204.506689:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x35:0x0](ffff8ab3d1f56c08) 00000080:00200008:0.0:1787827204.506694:0:48442:0:(file.c:1090:ll_file_open()) START file run_tar.sh:[0x240001b71:0x35:0x0](ffff8ab3d1f56c08), flags 100301 00000002:00100000:0.0:1787827204.506706:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3f07343c0 x1874671280540032/t17179869478(17179869478) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.506730:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc222c from it ffff8ab3d0a44f60 00010000:00010000:0.0:1787827204.506733:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.506740:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.506751:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x35:0x0](ffff8ab3d1f56c08), inode mode 81ed mask 40 00000080:00200008:0.0:1787827204.506761:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file run_tar.sh:[0x240001b71:0x35:0x0](ffff8ab3d1f56c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.506764:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x35:0x0](ffff8ab3d1f56c08),name=run_tar.sh 00000002:00010000:0.0:1787827204.506771:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x35:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.506777:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.506783:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.506788:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.506791:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x35:0x0](ffff8ab3d1f56c08) 00000080:00010000:0.0:1787827204.506792:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x35:0x0](ffff8ab3d1f56c08) for lock 0x268e1a602cc222c 00000080:00010000:0.0:1787827204.506795:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc222c from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.506797:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.506803:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.506811:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x35:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.506824:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x35:0x0] 00000008:00010000:0.0:1787827204.506832:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.506834:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f3b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.506853:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2233 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.506860:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2233 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.506880:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1787827204.507727:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x35:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787827204.507733:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.507739:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.507742:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f3b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.507747:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.507751:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f3b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.507760:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x35:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.507782:0:48442:0:(file.c:2652:do_file_write_iter()) START file run_tar.sh:[0x240001b71:0x35:0x0], ppos: 0, count: 1654 00000080:00200000:0.0:1787827204.507790:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 1654 bytes 00000080:00200000:0.0:1787827204.507793:0:48442:0:(file.c:2628:ll_do_tiny_write()) result: 0, original count 1654 00000080:00200000:0.0:1787827204.507796:0:48442:0:(file.c:2090:ll_file_io_generic()) run_tar.sh: write ppos: 0, bytes: 1654 00000080:00200000:0.0:1787827204.507799:0:48442:0:(file.c:1848:ll_io_set_mirror()) run_tar.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.507803:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@000000004e0306c0[0x110, 1, [0x240001b71:0x35:0x0] hash] 00000020:00200000:0.0:1787827204.507804:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 1654) 0 0 00000080:00200000:0.0:1787827204.507806:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x35:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.507811:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1787827204.507816:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1654) 1654, 1654 00020000:00200000:0.0:1787827204.507819:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.507828:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1787827204.507837:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.507840:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f3b0, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.507848:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3c7ca6800 type 11 mode 2 res 10737419264/155 (0 4095) 00010000:00010000:0.0:1787827204.507870:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.507878:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.507884:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.507912:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0737c00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280540288:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.507941:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4cb40 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.508612:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:0.0:1787827204.508643:0:48409:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294799360 00000100:00100000:0.0:1787827204.508649:0:48409:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294799360 00000100:00100000:0.0:1787827204.508678:0:48409:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 258 00000100:00100000:0.0:1787827204.508684:0:48409:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3e0ba5680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:77210:x1874671294799360:12345-192.168.203.124@tcp:104: 00010000:00010000:0.0:1787827204.508691:0:48409:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2233 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9f91 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.508727:0:48409:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787827204.508734:0:48409:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3e0ba5680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:77210:x1874671294799360:12345-192.168.203.124@tcp:104: Request processed in 50us (130us total) trans 0 rc 0/0 00000100:00100000:0.0:1787827204.508744:0:48409:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 258 00010000:00010000:3.0:1787827204.508748:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.508755:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2233 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9f91 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.508764:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6959e00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.508769:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2233 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9f91 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.508781:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x35:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.508788:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x35:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.508846:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2233 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9f91 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.508853:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.508869:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2233 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9f91 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.508876:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2233 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9f91 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.508904:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.508910:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.508914:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.509415:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0737c00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280540288:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.509429:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.509437:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.509444:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.509947:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.509963:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.509981:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.509991:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.509999:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.510005:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.510023:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) run_tar.sh: write [0, 1654) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.510031:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 1654 bytes 00000080:00200000:0.0:1787827204.510056:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 1654, copied 1654 00000080:00200000:0.0:1787827204.510062:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000991f54c6[3 00000000132192b6 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.510065:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000ff2e439b 17ffffc0001001 3:0 ffff8ab3d06ff488 0 lru 00000080:00200000:0.0:1787827204.510071:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000009f0e513a 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000f5856a27 00000000050f1b3f > 3< 0 0 > 4< 0 0 8 2887680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.510073:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000991f54c6 00000080:00200000:0.0:1787827204.510074:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.510077:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 1, from 0, to 1654 prio 0 00000008:00010000:0.0:1787827204.510087:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000003874a3b1 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.510110:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f56db8, count 1, dirtied 1 00000080:00200000:0.0:1787827204.510115:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 1 pages 1654 bytes, tot: 1654 00000080:00200000:0.0:1787827204.510118:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) run_tar.sh: write bytes 0, result: 1654 00000020:00010000:0.0:1787827204.510128:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.510131:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.510133:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f3b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.510137:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.510144:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.510151:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.510153:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f3b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.510157:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 1654 bytes 00000080:00200000:0.0:1787827204.510161:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1787827204.510165:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x35:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.510168:0:48442:0:(file.c:2240:ll_file_io_generic()) run_tar.sh: 2 io complete with rc: 0, result: 1654, restart: 0 00000080:00200000:0.0:1787827204.510172:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x35:0x0] updating i_size 1654 i_blocks 0 00000080:00200000:0.0:1787827204.510175:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 1654 00000080:00200008:0.0:1787827204.510190:0:48442:0:(file.c:479:ll_file_release()) START file run_tar.sh:[0x240001b71:0x35:0x0](ffff8ab3d1f56c08), flags 100001 00000002:00100000:0.0:1787827204.510221:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3f07343c0 x1874671280540032/t17179869478(17179869478) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.510232:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.510238:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3f0737c00 x1874671280540672/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.510253:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0737c00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280540672:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.510292:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4cb40 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.510589:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280540416:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787827204.511034:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0737c00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280540672:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.511047:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3f07343c0 x1874671280540032/t17179869478(17179869478) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.511075:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.511083:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.511091:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.511115:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.511123:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.511128:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.511131:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.511136:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.511138:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.511144:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.511151:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.511156:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.511159:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.511160:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.511162:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.511165:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=run_tar.sh, flags=8 00000080:00200008:0.0:1787827204.511169:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file run_tar.sh:[0x240001b71:0x35:0x0](ffff8ab3d1f56c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.511172:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x35:0x0](ffff8ab3d1f56c08),name=run_tar.sh 00000002:00010000:0.0:1787827204.511179:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x35:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.511184:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.511190:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.511195:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.511197:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x35:0x0](ffff8ab3d1f56c08) 00000080:00010000:0.0:1787827204.511199:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x35:0x0](ffff8ab3d1f56c08) for lock 0x268e1a602cc222c 00000080:00010000:0.0:1787827204.511201:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc222c from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.511203:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.511212:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.511221:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x35:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.511225:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x35:0x0] 00000008:00010000:0.0:1787827204.511232:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.511235:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f3b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.511244:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.511255:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.511265:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.511269:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.511279:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ad52bba1/0x268e1a602cc223a lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9f98 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.511290:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.511293:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x35:0x0] updating i_size 1654 i_blocks 0 00000020:00010000:0.0:1787827204.511296:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.511298:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.511300:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f3b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.511303:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.511306:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f3b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.511311:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x35:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.511337:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.511343:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.511348:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.511355:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.511360:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.511363:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.511365:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.511367:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.511370:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.511375:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.511380:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.511382:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.511385:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.511387:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.511389:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.511397:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-benchmark.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.511400:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-benchmark.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.511405:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-benchmark.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.511424:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cee00/0x268e1a602cc2241 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.511430:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cee00/0x268e1a602cc2241 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.511435:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cee00/0x268e1a602cc2241 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.511444:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afbc00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280540800:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.511475:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4cb40 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.512288:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afbc00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280540800:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.512296:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cee00/0x268e1a602cc2241 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.512303:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cee00/0x268e1a602cc2241 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.512307:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cee00/0x268e1a602cc2241 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.512311:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cee00/0x268e1a602cc2241 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.512316:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cee00/0x268e1a602cc2241 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.512320:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cee00 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.512322:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cee00/0x268e1a602cc2241 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.512330:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cee00/0x268e1a602cc2241 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.512335:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cee00/0x268e1a602cc2241 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.512339:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.512342:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cee00/0x268e1a602cc2241 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.512345:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cee00/0x268e1a602cc2241 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.512353:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0afbc00 x1874671280540800/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 592/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.512360:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.512368:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.512389:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.512394:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.512398:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.512404:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.512409:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.512412:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.512414:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.512417:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.512420:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.512425:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.512431:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.512434:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.512436:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.512438:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.512439:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.512446:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-benchmark.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.512449:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-benchmark.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.512454:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-benchmark.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.512468:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc2248 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.512473:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc2248 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.512478:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc2248 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.512486:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afbc00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280540928:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.512515:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4cb40 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.513741:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afbc00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280540928:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.513749:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc2248 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.513759:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc2248 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.513763:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc2248 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.513767:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc2248 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.513772:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc2248 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.513775:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07ce200 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.513778:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc2248 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.513781:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc2248 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.513785:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc2248 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.513790:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.513792:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc2248 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.513796:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc2248 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.513803:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0afbc00 x1874671280540928/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 592/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.513814:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.513823:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.513856:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.513861:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.513865:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.513872:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.513877:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.513879:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.513881:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.513883:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.513905:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.513911:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.513919:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.513924:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.513929:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.513932:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.513936:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.513944:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.513949:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=sanity-benchmark.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be2a80, open_flags 80c1, mode 81a4 opened 0 00000080:00200000:0.0:1787827204.513956:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-benchmark.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.513969:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-benchmark.sh,[0x240001b71:0x36:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.513992:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.513998:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.514003:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.514007:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afbc00 x1874671280541184/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.514013:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.514022:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afbc00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280541184:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.514088:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4cb40 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.515208:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afbc00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280541184:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.515217:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.515230:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.515237:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.515247:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.515259:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.515271:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.515280:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.515285:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.515292:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3d0afbc00 x1874671280541184/t17179869480(17179869480) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 664/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.515299:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0afbc00 x1874671280541184/t17179869480(17179869480) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 664/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.515307:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.515315:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.515321:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/54 (0 0) 00000080:00200000:0.0:1787827204.515325:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.515330:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=sanity-benchmark.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.515343:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x36:0x0](ffff8ab3d1f53a88) 00000080:00200000:0.0:1787827204.515382:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x36:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.515386:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x36:0x0](ffff8ab3d1f53a88) 00000080:00010000:0.0:1787827204.515388:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x36:0x0](ffff8ab3d1f53a88) for lock 0x268e1a602cc224f 00000080:00010000:0.0:1787827204.515393:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x36:0x0](ffff8ab3d1f53a88) 00000080:00200008:0.0:1787827204.515398:0:48442:0:(file.c:1090:ll_file_open()) START file sanity-benchmark.sh:[0x240001b71:0x36:0x0](ffff8ab3d1f53a88), flags 100301 00000002:00100000:0.0:1787827204.515408:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0afbc00 x1874671280541184/t17179869480(17179869480) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 592/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.515429:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc224f from it ffff8ab3d0a44f60 00010000:00010000:0.0:1787827204.515433:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.515439:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.515448:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x36:0x0](ffff8ab3d1f53a88), inode mode 81a4 mask 40 00000080:00200008:0.0:1787827204.515456:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-benchmark.sh:[0x240001b71:0x36:0x0](ffff8ab3d1f53a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.515459:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x36:0x0](ffff8ab3d1f53a88),name=sanity-benchmark.sh 00000002:00010000:0.0:1787827204.515467:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x36:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.515472:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.515478:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.515483:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.515485:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x36:0x0](ffff8ab3d1f53a88) 00000080:00010000:0.0:1787827204.515487:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x36:0x0](ffff8ab3d1f53a88) for lock 0x268e1a602cc224f 00000080:00010000:0.0:1787827204.515490:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc224f from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.515493:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.515498:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.515505:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x36:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.515509:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x36:0x0] 00000008:00010000:0.0:1787827204.515514:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.515516:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f330, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.515534:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.515540:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.515560:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1787827204.515578:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787827204.515590:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280541312:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.516342:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.516358:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9fec expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.516367:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d9fec expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.516375:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d9fec expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.516383:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9fec expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.516390:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.516399:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9fec expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.516405:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9fec expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.516416:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0d9fec expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.516422:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x36:0x0] updating i_size 0 i_blocks 0 00010000:00010000:3.0:1787827204.516423:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d9fec expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.516426:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.516434:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.516438:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f330 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000100:00100000:3.0:1787827204.516441:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280541312:192.168.203.124@tcp:101:cp.0 00000020:00010000:0.0:1787827204.516443:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.516449:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f330 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.516459:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x36:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.516479:0:48442:0:(file.c:2652:do_file_write_iter()) START file sanity-benchmark.sh:[0x240001b71:0x36:0x0], ppos: 0, count: 6548 00000080:00200000:0.0:1787827204.516484:0:48442:0:(file.c:2090:ll_file_io_generic()) sanity-benchmark.sh: write ppos: 0, bytes: 6548 00000080:00200000:0.0:1787827204.516487:0:48442:0:(file.c:1848:ll_io_set_mirror()) sanity-benchmark.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.516491:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@00000000510956ab[0x110, 1, [0x240001b71:0x36:0x0] hash] 00000020:00200000:0.0:1787827204.516493:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 6548) 0 0 00000080:00200000:0.0:1787827204.516495:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x36:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.516499:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 1] 00020000:00200000:0.0:1787827204.516502:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 6548) 6548, 6548 00020000:00200000:0.0:1787827204.516505:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.516511:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 1] 00000008:00010000:0.0:1787827204.516516:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.516518:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f330, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.516524:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d2d3cc00 type 11 mode 2 res 11811161088/156 (0 8191) 00010000:00010000:0.0:1787827204.516539:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.516545:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.516551:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.516561:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af8b40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280541440:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.516590:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ccc0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.517337:0:48409:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294799616 00000100:00100000:0.0:1787827204.517342:0:48409:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294799616 00000100:00100000:0.0:1787827204.517372:0:48409:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 260 00000100:00100000:0.0:1787827204.517377:0:48409:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22ff0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:82330:x1874671294799616:12345-192.168.203.124@tcp:104: 00010000:00010000:0.0:1787827204.517384:0:48409:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9fec expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.517419:0:48409:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787827204.517425:0:48409:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22ff0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:82330:x1874671294799616:12345-192.168.203.124@tcp:104: Request processed in 47us (133us total) trans 0 rc 0/0 00000100:00100000:0.0:1787827204.517432:0:48409:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 260 00010000:00010000:3.0:1787827204.517437:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.517445:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0d9fec expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.517457:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cf400 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.517463:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0d9fec expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.517477:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x36:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.517485:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x36:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.517565:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9fec expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.517572:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.517589:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9fec expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.517600:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc2256 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0d9fec expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.517610:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.517613:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.517615:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.518000:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af8b40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280541440:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.518011:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.518018:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.518024:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.518788:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.518796:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.518805:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.518816:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.518835:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.518851:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.518867:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) sanity-benchmark.sh: write [0, 6548) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.518875:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.518931:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:0.0:1787827204.518940:0:48442:0:(rw26.c:933:ll_write_end()) page@000000003d57f83b[3 00000000b8660537 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.518944:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000009380de9e 17ffffc0001001 3:0 ffff8ab3d06ff3a0 0 lru 00000080:00200000:0.0:1787827204.518953:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e11905cc 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000006a588f83 00000000eba9de99 > 3< 0 0 > 4< 0 0 8 3563520 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.518956:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000003d57f83b 00000080:00200000:0.0:1787827204.518958:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.518962:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 1 of 0 to 2452 bytes 00000080:00200000:0.0:1787827204.518989:0:48442:0:(rw26.c:875:ll_write_end()) pos 4096, len 2452, copied 2452 00000080:00200000:0.0:1787827204.518993:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000b0a38286[3 00000000b8660537 2 1 000000002663f4b0 0 0 1] 00000080:00200000:0.0:1787827204.518996:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000004d07d3f5 17ffffc0001001 3:0 ffff8ab3d06ff9f8 1 lru 00000080:00200000:0.0:1787827204.519001:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e5324774 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000006a588f83 00000000eba9de99 > 3< 0 0 > 4< 0 0 8 3563520 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.519004:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000b0a38286 00000080:00200000:0.0:1787827204.519005:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 2. 00000080:00200000:0.0:1787827204.519007:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 2, from 0, to 2452 prio 0 00000008:00010000:0.0:1787827204.519018:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 0000000038abb89c ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.519030:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000038abb89c ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.519038:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f53c38, count 2, dirtied 2 00000080:00200000:0.0:1787827204.519041:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 2 pages 6548 bytes, tot: 6548 00000080:00200000:0.0:1787827204.519043:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) sanity-benchmark.sh: write bytes 0, result: 6548 00000020:00010000:0.0:1787827204.519051:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.519054:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.519056:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f330 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.519060:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.519066:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.519073:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.519075:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f330 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.519079:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 6548 bytes 00000080:00200000:0.0:1787827204.519081:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 1] 00000080:00200000:0.0:1787827204.519085:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x36:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.519087:0:48442:0:(file.c:2240:ll_file_io_generic()) sanity-benchmark.sh: 2 io complete with rc: 0, result: 6548, restart: 0 00000080:00200000:0.0:1787827204.519091:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x36:0x0] updating i_size 6548 i_blocks 0 00000080:00200000:0.0:1787827204.519094:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 6548 00000080:00200008:0.0:1787827204.519121:0:48442:0:(file.c:479:ll_file_release()) START file sanity-benchmark.sh:[0x240001b71:0x36:0x0](ffff8ab3d1f53a88), flags 100001 00000002:00100000:0.0:1787827204.519147:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3d0afbc00 x1874671280541184/t17179869480(17179869480) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 592/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.519155:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.519159:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af8b40 x1874671280541824/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.519169:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af8b40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280541824:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.519198:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ccc0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.519955:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af8b40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280541824:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.519974:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3d0afbc00 x1874671280541184/t17179869480(17179869480) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 592/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.520011:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.520022:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.520033:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.520046:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.520054:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.520059:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.520063:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.520068:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.520072:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.520081:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.520092:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.520113:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.520118:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.520122:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.520126:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.520130:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-benchmark.sh, flags=8 00000080:00200008:0.0:1787827204.520135:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-benchmark.sh:[0x240001b71:0x36:0x0](ffff8ab3d1f53a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.520140:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x36:0x0](ffff8ab3d1f53a88),name=sanity-benchmark.sh 00000002:00010000:0.0:1787827204.520151:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x36:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.520159:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.520170:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.520178:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.520183:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x36:0x0](ffff8ab3d1f53a88) 00000080:00010000:0.0:1787827204.520186:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x36:0x0](ffff8ab3d1f53a88) for lock 0x268e1a602cc224f 00000080:00010000:0.0:1787827204.520191:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc224f from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.520194:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.520203:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.520217:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x36:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.520223:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x36:0x0] 00000008:00010000:0.0:1787827204.520232:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.520235:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f330, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.520245:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.520258:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.520269:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.520273:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.520284:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000082e7976e/0x268e1a602cc225d lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9ff3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.520295:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.520301:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x36:0x0] updating i_size 6548 i_blocks 0 00000020:00010000:0.0:1787827204.520305:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.520309:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.520313:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f330 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.520317:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.520320:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f330 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.520327:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x36:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.520360:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.520370:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.520378:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.520389:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.520398:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.520402:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.520406:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.520411:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.520415:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.520432:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.520442:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.520446:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.520450:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.520453:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.520456:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.520468:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-compr.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.520472:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-compr.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.520482:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-compr.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.520507:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2264 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.520516:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2264 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.520525:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2264 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.520541:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afa1c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280541952:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.520581:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ccc0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.521400:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afa1c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280541952:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.521411:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2264 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.521420:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2264 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.521427:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2264 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.521433:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2264 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.521441:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2264 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.521447:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07ce600 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.521451:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2264 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.521458:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2264 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.521464:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2264 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.521471:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.521475:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2264 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.521482:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2264 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.521494:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0afa1c0 x1874671280541952/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.521504:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.521515:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.521540:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.521547:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.521554:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.521564:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.521571:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.521575:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.521578:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.521582:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.521586:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.521594:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.521603:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.521607:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.521611:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.521614:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.521617:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.521624:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-compr.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.521627:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-compr.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.521635:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-compr.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.521654:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc226b lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.521666:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc226b lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.521673:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc226b lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.521686:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afa1c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280542080:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.521723:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ccc0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.522659:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afa1c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280542080:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.522671:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc226b lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.522679:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc226b lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.522687:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc226b lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.522694:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc226b lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.522701:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc226b lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.522707:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cea00 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.522711:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc226b lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.522719:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc226b lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.522726:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc226b lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.522733:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.522737:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc226b lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.522745:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc226b lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.522758:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0afa1c0 x1874671280542080/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.522770:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.522781:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.522823:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.522832:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.522840:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.522851:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.522859:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.522863:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.522866:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.522871:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.522874:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.522921:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.522933:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.522939:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.522945:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.522949:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.522954:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.522964:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.522968:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=sanity-compr.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be3080, open_flags 80c1, mode 81a4 opened 0 00000080:00200000:0.0:1787827204.522975:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-compr.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.522987:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-compr.sh,[0x240001b71:0x37:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.523013:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.523022:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.523031:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.523037:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afa1c0 x1874671280542336/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.523047:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.523062:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afa1c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280542336:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.523152:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ccc0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.524407:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afa1c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280542336:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.524421:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.524450:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.524460:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.524470:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.524482:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.524494:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.524503:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.524511:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.524522:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3d0afa1c0 x1874671280542336/t17179869482(17179869482) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.524535:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0afa1c0 x1874671280542336/t17179869482(17179869482) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.524548:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.524560:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.524570:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/55 (0 0) 00000080:00200000:0.0:1787827204.524574:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.524581:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=sanity-compr.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.524598:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x37:0x0](ffff8ab3d1f55348) 00000080:00200000:0.0:1787827204.524650:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x37:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.524656:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x37:0x0](ffff8ab3d1f55348) 00000080:00010000:0.0:1787827204.524659:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x37:0x0](ffff8ab3d1f55348) for lock 0x268e1a602cc2272 00000080:00010000:0.0:1787827204.524667:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x37:0x0](ffff8ab3d1f55348) 00000080:00200008:0.0:1787827204.524673:0:48442:0:(file.c:1090:ll_file_open()) START file sanity-compr.sh:[0x240001b71:0x37:0x0](ffff8ab3d1f55348), flags 100301 00000002:00100000:0.0:1787827204.524688:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0afa1c0 x1874671280542336/t17179869482(17179869482) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.524717:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2272 from it ffff8ab3d0a44f60 00010000:00010000:0.0:1787827204.524722:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.524730:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.524744:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x37:0x0](ffff8ab3d1f55348), inode mode 81a4 mask 40 00000080:00200008:0.0:1787827204.524755:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-compr.sh:[0x240001b71:0x37:0x0](ffff8ab3d1f55348), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.524760:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x37:0x0](ffff8ab3d1f55348),name=sanity-compr.sh 00000002:00010000:0.0:1787827204.524770:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x37:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.524778:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.524788:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.524796:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.524801:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x37:0x0](ffff8ab3d1f55348) 00000080:00010000:0.0:1787827204.524804:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x37:0x0](ffff8ab3d1f55348) for lock 0x268e1a602cc2272 00000080:00010000:0.0:1787827204.524808:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2272 from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.524812:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.524821:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.524834:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x37:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.524840:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x37:0x0] 00000008:00010000:0.0:1787827204.524850:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.524852:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f6b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.524875:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001735bf49/0x268e1a602cc2279 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.524882:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001735bf49/0x268e1a602cc2279 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.524931:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1787827204.525766:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x37:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787827204.525771:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.525776:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.525782:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f6b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.525785:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.525787:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f6b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.525794:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x37:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.525814:0:48442:0:(file.c:2652:do_file_write_iter()) START file sanity-compr.sh:[0x240001b71:0x37:0x0], ppos: 0, count: 7803 00000080:00200000:0.0:1787827204.525818:0:48442:0:(file.c:2090:ll_file_io_generic()) sanity-compr.sh: write ppos: 0, bytes: 7803 00000080:00200000:0.0:1787827204.525822:0:48442:0:(file.c:1848:ll_io_set_mirror()) sanity-compr.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.525826:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@0000000047b1d265[0x110, 1, [0x240001b71:0x37:0x0] hash] 00000020:00200000:0.0:1787827204.525827:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 7803) 0 0 00000080:00200000:0.0:1787827204.525829:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x37:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.525834:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 1] 00020000:00200000:0.0:1787827204.525838:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 7803) 7803, 7803 00020000:00200000:0.0:1787827204.525841:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.525849:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 1] 00000008:00010000:0.0:1787827204.525854:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.525857:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f6b0, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.525863:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3c7ca6800 type 11 mode 2 res 10737419264/156 (0 8191) 00010000:00010000:0.0:1787827204.525878:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.525904:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.525911:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.525924:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afa940 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280542592:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.525955:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c300 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.526673:0:48409:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294799872 00000100:00100000:0.0:1787827204.526678:0:48409:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294799872 00000100:00100000:0.0:1787827204.526711:0:48409:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 262 00000100:00100000:0.0:1787827204.526718:0:48409:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22fe580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:77209:x1874671294799872:12345-192.168.203.124@tcp:104: 00010000:00010000:0.0:1787827204.526725:0:48409:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001735bf49/0x268e1a602cc2279 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da047 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.526760:0:48409:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787827204.526765:0:48409:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22fe580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:77209:x1874671294799872:12345-192.168.203.124@tcp:104: Request processed in 48us (128us total) trans 0 rc 0/0 00000100:00100000:0.0:1787827204.526771:0:48409:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 262 00010000:00010000:3.0:1787827204.526784:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.526793:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001735bf49/0x268e1a602cc2279 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da047 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.526805:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07ce200 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.526811:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001735bf49/0x268e1a602cc2279 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da047 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.526825:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x37:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.526832:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x37:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.526901:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001735bf49/0x268e1a602cc2279 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da047 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.526908:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.526926:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001735bf49/0x268e1a602cc2279 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da047 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.526933:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001735bf49/0x268e1a602cc2279 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da047 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.526958:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.526962:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.526966:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.527397:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afa940 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280542592:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.527409:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.527417:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.527425:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.527960:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.527975:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.527998:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.528009:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.528024:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.528031:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.528048:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) sanity-compr.sh: write [0, 7803) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.528057:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.528085:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:0.0:1787827204.528092:0:48442:0:(rw26.c:933:ll_write_end()) page@000000008f39a2a9[3 00000000b8a55100 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.528108:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000009a2c71d8 17ffffc0001001 3:0 ffff8ab3d06ff0e8 0 lru 00000080:00200000:0.0:1787827204.528115:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000fd78e621 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000f5856a27 00000000969aeff8 > 3< 0 0 > 4< 0 0 8 2859008 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.528116:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000008f39a2a9 00000080:00200000:0.0:1787827204.528118:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.528120:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 1 of 0 to 3707 bytes 00000080:00200000:0.0:1787827204.528141:0:48442:0:(rw26.c:875:ll_write_end()) pos 4096, len 3707, copied 3707 00000080:00200000:0.0:1787827204.528145:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000090169b97[3 00000000b8a55100 2 1 000000002663f4b0 0 0 1] 00000080:00200000:0.0:1787827204.528148:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000004aa96fc 17ffffc0001001 3:0 ffff8ab3d06ff740 1 lru 00000080:00200000:0.0:1787827204.528153:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000be9af077 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000f5856a27 00000000969aeff8 > 3< 0 0 > 4< 0 0 8 2859008 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.528154:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000090169b97 00000080:00200000:0.0:1787827204.528155:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 2. 00000080:00200000:0.0:1787827204.528158:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 2, from 0, to 3707 prio 0 00000008:00010000:0.0:1787827204.528172:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000f2fe9661 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.528185:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000f2fe9661 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.528194:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f554f8, count 2, dirtied 2 00000080:00200000:0.0:1787827204.528198:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 2 pages 7803 bytes, tot: 7803 00000080:00200000:0.0:1787827204.528199:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) sanity-compr.sh: write bytes 0, result: 7803 00000020:00010000:0.0:1787827204.528209:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.528212:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.528214:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f6b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.528218:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.528226:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.528253:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.528258:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f6b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.528264:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 7803 bytes 00000080:00200000:0.0:1787827204.528269:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 1] 00000080:00200000:0.0:1787827204.528276:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x37:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.528281:0:48442:0:(file.c:2240:ll_file_io_generic()) sanity-compr.sh: 2 io complete with rc: 0, result: 7803, restart: 0 00000080:00200000:0.0:1787827204.528287:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x37:0x0] updating i_size 7803 i_blocks 0 00000080:00200000:0.0:1787827204.528291:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 7803 00000080:00200008:0.0:1787827204.528316:0:48442:0:(file.c:479:ll_file_release()) START file sanity-compr.sh:[0x240001b71:0x37:0x0](ffff8ab3d1f55348), flags 100001 00000002:00100000:0.0:1787827204.528349:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3d0afa1c0 x1874671280542336/t17179869482(17179869482) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.528358:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.528361:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afa940 x1874671280542976/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.528372:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afa940 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280542976:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.528404:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c300 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.528690:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280542720:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787827204.529171:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afa940 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280542976:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.529186:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3d0afa1c0 x1874671280542336/t17179869482(17179869482) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.529215:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.529224:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.529232:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.529240:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.529245:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.529249:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.529251:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.529255:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.529257:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.529267:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.529275:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.529279:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.529282:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.529284:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.529287:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.529289:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-compr.sh, flags=8 00000080:00200008:0.0:1787827204.529292:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-compr.sh:[0x240001b71:0x37:0x0](ffff8ab3d1f55348), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.529296:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x37:0x0](ffff8ab3d1f55348),name=sanity-compr.sh 00000002:00010000:0.0:1787827204.529302:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x37:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.529307:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.529313:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.529317:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.529319:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x37:0x0](ffff8ab3d1f55348) 00000080:00010000:0.0:1787827204.529321:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x37:0x0](ffff8ab3d1f55348) for lock 0x268e1a602cc2272 00000080:00010000:0.0:1787827204.529323:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2272 from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.529325:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.529330:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.529339:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x37:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.529343:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x37:0x0] 00000008:00010000:0.0:1787827204.529349:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.529351:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f6b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.529358:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.529365:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.529372:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.529374:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.529380:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f442d85/0x268e1a602cc2280 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da04e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.529385:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.529388:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x37:0x0] updating i_size 7803 i_blocks 0 00000020:00010000:0.0:1787827204.529391:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.529393:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.529395:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f6b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.529397:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.529398:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f6b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.529403:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x37:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.529428:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.529434:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.529439:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.529448:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.529456:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.529460:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.529463:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.529468:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.529472:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.529481:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.529491:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.529509:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.529512:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.529515:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.529516:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.529527:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-dom.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.529530:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-dom.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.529537:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-dom.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.529558:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc2287 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.529563:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc2287 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.529570:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc2287 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.529579:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af8000 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280543104:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.529614:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c300 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.530462:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af8000 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280543104:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.530471:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc2287 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.530479:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc2287 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.530484:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc2287 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.530489:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc2287 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.530495:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc2287 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.530501:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cf600 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.530505:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc2287 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.530512:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc2287 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.530519:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc2287 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.530527:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.530532:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc2287 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.530540:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc2287 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.530553:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0af8000 x1874671280543104/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.530562:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.530573:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.530595:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.530601:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.530606:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.530613:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.530619:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.530621:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.530624:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.530627:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.530629:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.530634:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.530643:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.530646:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.530649:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.530651:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.530653:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.530670:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-dom.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.530673:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-dom.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.530679:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-dom.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.530694:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc228e lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.530699:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc228e lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.530705:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc228e lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.530717:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af8000 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280543232:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.530765:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c300 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.531600:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af8000 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280543232:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.531609:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc228e lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.531616:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc228e lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.531621:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc228e lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.531626:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc228e lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.531631:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc228e lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.531635:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cea00 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.531637:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc228e lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.531641:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc228e lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.531645:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc228e lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.531649:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.531652:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc228e lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.531657:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc228e lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.531665:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0af8000 x1874671280543232/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.531681:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.531691:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.531734:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.531741:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.531746:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.531754:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.531759:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.531762:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.531764:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.531768:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.531770:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.531775:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.531781:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.531785:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.531788:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.531790:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.531792:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.531798:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.531802:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=sanity-dom.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be2a80, open_flags 80c1, mode 81a4 opened 0 00000080:00200000:0.0:1787827204.531807:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-dom.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.531817:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-dom.sh,[0x240001b71:0x38:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.531838:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.531843:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.531850:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.531855:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af8000 x1874671280543488/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.531861:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.531870:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af8000 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280543488:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.531968:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c300 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.533208:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af8000 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280543488:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.533220:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.533232:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.533238:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.533245:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.533251:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.533257:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.533267:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.533272:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.533278:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3d0af8000 x1874671280543488/t17179869484(17179869484) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.533285:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0af8000 x1874671280543488/t17179869484(17179869484) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.533293:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.533301:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.533307:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/56 (0 0) 00000080:00200000:0.0:1787827204.533310:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.533316:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=sanity-dom.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.533331:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x38:0x0](ffff8ab3d1f57448) 00000080:00200000:0.0:1787827204.533378:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x38:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.533382:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x38:0x0](ffff8ab3d1f57448) 00000080:00010000:0.0:1787827204.533385:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x38:0x0](ffff8ab3d1f57448) for lock 0x268e1a602cc2295 00000080:00010000:0.0:1787827204.533390:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x38:0x0](ffff8ab3d1f57448) 00000080:00200008:0.0:1787827204.533396:0:48442:0:(file.c:1090:ll_file_open()) START file sanity-dom.sh:[0x240001b71:0x38:0x0](ffff8ab3d1f57448), flags 100301 00000002:00100000:0.0:1787827204.533408:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0af8000 x1874671280543488/t17179869484(17179869484) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.533433:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2295 from it ffff8ab3d0a44f60 00010000:00010000:0.0:1787827204.533437:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.533444:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.533455:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x38:0x0](ffff8ab3d1f57448), inode mode 81a4 mask 40 00000080:00200008:0.0:1787827204.533464:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-dom.sh:[0x240001b71:0x38:0x0](ffff8ab3d1f57448), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.533467:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x38:0x0](ffff8ab3d1f57448),name=sanity-dom.sh 00000002:00010000:0.0:1787827204.533475:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x38:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.533480:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.533487:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.533492:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.533495:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x38:0x0](ffff8ab3d1f57448) 00000080:00010000:0.0:1787827204.533497:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x38:0x0](ffff8ab3d1f57448) for lock 0x268e1a602cc2295 00000080:00010000:0.0:1787827204.533499:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2295 from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.533502:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.533507:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.533516:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x38:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.533521:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x38:0x0] 00000008:00010000:0.0:1787827204.533527:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.533529:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f930, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.533549:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.533557:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.533577:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1787827204.533599:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787827204.533611:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280543616:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.534358:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.534370:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da0a2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.534377:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da0a2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.534385:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da0a2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.534393:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da0a2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.534409:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.534416:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da0a2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.534428:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da0a2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.534433:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da0a2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.534439:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da0a2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.534441:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x38:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787827204.534448:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.534451:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000100:00100000:3.0:1787827204.534455:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280543616:192.168.203.124@tcp:101:cp.0 00000020:00010000:0.0:1787827204.534456:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f930 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.534464:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.534467:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f930 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.534474:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x38:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.534497:0:48442:0:(file.c:2652:do_file_write_iter()) START file sanity-dom.sh:[0x240001b71:0x38:0x0], ppos: 0, count: 6821 00000080:00200000:0.0:1787827204.534501:0:48442:0:(file.c:2090:ll_file_io_generic()) sanity-dom.sh: write ppos: 0, bytes: 6821 00000080:00200000:0.0:1787827204.534505:0:48442:0:(file.c:1848:ll_io_set_mirror()) sanity-dom.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.534509:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@00000000bdc96cee[0x110, 1, [0x240001b71:0x38:0x0] hash] 00000020:00200000:0.0:1787827204.534510:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 6821) 0 0 00000080:00200000:0.0:1787827204.534512:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x38:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.534517:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 1] 00020000:00200000:0.0:1787827204.534521:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 6821) 6821, 6821 00020000:00200000:0.0:1787827204.534524:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.534533:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 1] 00000008:00010000:0.0:1787827204.534539:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.534542:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f930, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.534548:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d2d3cc00 type 11 mode 2 res 11811161088/157 (0 8191) 00010000:00010000:0.0:1787827204.534568:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.534575:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.534581:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.534590:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afb480 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280543744:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.534620:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ce40 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.535314:0:48409:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294800128 00000100:00100000:0.0:1787827204.535319:0:48409:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294800128 00000100:00100000:0.0:1787827204.535351:0:48409:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 264 00000100:00100000:0.0:1787827204.535357:0:48409:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3d22ff0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:77210:x1874671294800128:12345-192.168.203.124@tcp:104: 00010000:00010000:0.0:1787827204.535365:0:48409:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da0a2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.535402:0:48409:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787827204.535408:0:48409:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3d22ff0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:77210:x1874671294800128:12345-192.168.203.124@tcp:104: Request processed in 51us (131us total) trans 0 rc 0/0 00000100:00100000:0.0:1787827204.535415:0:48409:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 264 00010000:00010000:3.0:1787827204.535424:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.535431:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da0a2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.535440:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cf200 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.535445:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da0a2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.535456:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x38:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.535463:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x38:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.535520:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da0a2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.535531:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.535551:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da0a2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.535560:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d2089f23/0x268e1a602cc229c lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da0a2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.535579:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.535583:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.535586:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.536070:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afb480 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280543744:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.536081:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.536093:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.536113:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.536415:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.536424:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.536448:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.536461:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.536473:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.536482:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.536500:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) sanity-dom.sh: write [0, 6821) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.536511:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.536541:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:0.0:1787827204.536551:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000ef7f6462[3 00000000dafae6bb 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.536555:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000006592bd52 17ffffc0001001 3:0 ffff8ab3d06ff570 0 lru 00000080:00200000:0.0:1787827204.536564:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000386792e5 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000006a588f83 00000000d41e0d32 > 3< 0 0 > 4< 0 0 8 3530752 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.536566:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000ef7f6462 00000080:00200000:0.0:1787827204.536568:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.536570:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 1 of 0 to 2725 bytes 00000080:00200000:0.0:1787827204.536602:0:48442:0:(rw26.c:875:ll_write_end()) pos 4096, len 2725, copied 2725 00000080:00200000:0.0:1787827204.536606:0:48442:0:(rw26.c:933:ll_write_end()) page@000000001894b5c2[3 00000000dafae6bb 2 1 000000002663f4b0 0 0 1] 00000080:00200000:0.0:1787827204.536608:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000061ed5180 17ffffc0001021 2:0 ffff8ab3d06ff1d0 1 lru 00000080:00200000:0.0:1787827204.536613:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000004971bf48 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000006a588f83 00000000d41e0d32 > 3< 0 0 > 4< 0 0 8 3530752 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.536614:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000001894b5c2 00000080:00200000:0.0:1787827204.536615:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 2. 00000080:00200000:0.0:1787827204.536618:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 2, from 0, to 2725 prio 0 00000008:00010000:0.0:1787827204.536630:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000f4293618 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.536642:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000f4293618 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.536651:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f575f8, count 2, dirtied 2 00000080:00200000:0.0:1787827204.536654:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 2 pages 6821 bytes, tot: 6821 00000080:00200000:0.0:1787827204.536656:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) sanity-dom.sh: write bytes 0, result: 6821 00000020:00010000:0.0:1787827204.536664:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.536667:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.536679:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f930 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.536682:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.536694:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.536701:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.536704:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f930 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.536708:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 6821 bytes 00000080:00200000:0.0:1787827204.536710:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 1] 00000080:00200000:0.0:1787827204.536715:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x38:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.536718:0:48442:0:(file.c:2240:ll_file_io_generic()) sanity-dom.sh: 2 io complete with rc: 0, result: 6821, restart: 0 00000080:00200000:0.0:1787827204.536722:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x38:0x0] updating i_size 6821 i_blocks 0 00000080:00200000:0.0:1787827204.536729:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 6821 00000080:00200008:0.0:1787827204.536743:0:48442:0:(file.c:479:ll_file_release()) START file sanity-dom.sh:[0x240001b71:0x38:0x0](ffff8ab3d1f57448), flags 100001 00000002:00100000:0.0:1787827204.536768:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3d0af8000 x1874671280543488/t17179869484(17179869484) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.536776:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.536779:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afb480 x1874671280544128/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.536790:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afb480 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280544128:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.536819:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ce40 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.537600:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afb480 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280544128:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.537619:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3d0af8000 x1874671280543488/t17179869484(17179869484) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.537648:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.537657:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.537666:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.537674:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.537680:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.537683:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.537685:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.537689:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.537691:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.537697:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.537704:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.537708:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.537711:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.537713:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.537714:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.537716:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-dom.sh, flags=8 00000080:00200008:0.0:1787827204.537720:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-dom.sh:[0x240001b71:0x38:0x0](ffff8ab3d1f57448), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.537723:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x38:0x0](ffff8ab3d1f57448),name=sanity-dom.sh 00000002:00010000:0.0:1787827204.537729:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x38:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.537733:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.537744:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.537748:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.537750:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x38:0x0](ffff8ab3d1f57448) 00000080:00010000:0.0:1787827204.537752:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x38:0x0](ffff8ab3d1f57448) for lock 0x268e1a602cc2295 00000080:00010000:0.0:1787827204.537754:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2295 from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.537756:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.537761:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.537770:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x38:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.537774:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x38:0x0] 00000008:00010000:0.0:1787827204.537780:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.537782:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f930, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.537789:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.537796:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.537803:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.537805:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.537815:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.537821:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.537824:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x38:0x0] updating i_size 6821 i_blocks 0 00000020:00010000:0.0:1787827204.537827:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.537829:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.537830:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f930 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.537832:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.537834:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f930 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.537839:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x38:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.537862:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.537868:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.537872:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.537878:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.537883:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.537915:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.537919:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.537924:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.537928:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.537944:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.537952:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.537955:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.537958:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.537959:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.537961:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.537969:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-ec.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.537972:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-ec.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.537978:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-ec.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.537997:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce800/0x268e1a602cc22aa lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.538002:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce800/0x268e1a602cc22aa lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.538008:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce800/0x268e1a602cc22aa lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.538017:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afb840 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280544256:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.538047:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ce40 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.538973:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afb840 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280544256:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.538984:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce800/0x268e1a602cc22aa lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.538992:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce800/0x268e1a602cc22aa lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.538997:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce800/0x268e1a602cc22aa lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.539002:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce800/0x268e1a602cc22aa lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.539006:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce800/0x268e1a602cc22aa lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.539010:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07ce800 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.539013:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce800/0x268e1a602cc22aa lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.539017:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce800/0x268e1a602cc22aa lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.539021:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce800/0x268e1a602cc22aa lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.539025:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.539029:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce800/0x268e1a602cc22aa lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.539033:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce800/0x268e1a602cc22aa lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.539042:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0afb840 x1874671280544256/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.539049:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.539059:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.539081:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.539088:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.539093:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.539114:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.539120:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.539123:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.539125:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.539129:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.539131:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.539137:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.539143:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.539147:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.539149:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.539151:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.539153:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.539158:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-ec.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.539161:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-ec.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.539167:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-ec.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.539182:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce400/0x268e1a602cc22b1 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.539198:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce400/0x268e1a602cc22b1 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.539203:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce400/0x268e1a602cc22b1 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.539212:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afb840 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280544384:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.539242:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ce40 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.540193:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afb840 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280544384:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.540205:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce400/0x268e1a602cc22b1 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.540215:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce400/0x268e1a602cc22b1 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.540222:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce400/0x268e1a602cc22b1 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.540229:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce400/0x268e1a602cc22b1 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.540237:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce400/0x268e1a602cc22b1 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.540244:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07ce400 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.540247:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce400/0x268e1a602cc22b1 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.540253:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce400/0x268e1a602cc22b1 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.540261:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce400/0x268e1a602cc22b1 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.540268:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.540273:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce400/0x268e1a602cc22b1 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.540290:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce400/0x268e1a602cc22b1 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.540304:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0afb840 x1874671280544384/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.540316:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.540328:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.540375:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.540385:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.540393:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.540404:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.540412:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.540417:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.540420:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.540424:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.540428:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.540443:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.540453:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.540458:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.540462:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.540466:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.540469:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.540478:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.540482:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=sanity-ec.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be3080, open_flags 80c1, mode 81a4 opened 0 00000080:00200000:0.0:1787827204.540489:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-ec.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.540503:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-ec.sh,[0x240001b71:0x39:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.540530:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.540538:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.540546:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.540553:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afb840 x1874671280544640/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.540563:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.540577:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afb840 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280544640:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.540656:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ce40 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.541950:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afb840 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280544640:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.541964:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.541981:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.541990:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.542002:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.542013:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.542025:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.542034:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.542044:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.542056:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3d0afb840 x1874671280544640/t17179869486(17179869486) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.542070:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0afb840 x1874671280544640/t17179869486(17179869486) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.542085:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.542113:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.542124:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/57 (0 0) 00000080:00200000:0.0:1787827204.542130:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.542139:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=sanity-ec.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.542161:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x39:0x0](ffff8ab3d1f500c8) 00000080:00200000:0.0:1787827204.542237:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x39:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.542244:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x39:0x0](ffff8ab3d1f500c8) 00000080:00010000:0.0:1787827204.542247:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x39:0x0](ffff8ab3d1f500c8) for lock 0x268e1a602cc22b8 00000080:00010000:0.0:1787827204.542255:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x39:0x0](ffff8ab3d1f500c8) 00000080:00200008:0.0:1787827204.542263:0:48442:0:(file.c:1090:ll_file_open()) START file sanity-ec.sh:[0x240001b71:0x39:0x0](ffff8ab3d1f500c8), flags 100301 00000002:00100000:0.0:1787827204.542282:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0afb840 x1874671280544640/t17179869486(17179869486) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.542315:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc22b8 from it ffff8ab3d0a44f60 00010000:00010000:0.0:1787827204.542321:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.542332:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.542348:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x39:0x0](ffff8ab3d1f500c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1787827204.542361:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-ec.sh:[0x240001b71:0x39:0x0](ffff8ab3d1f500c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.542367:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x39:0x0](ffff8ab3d1f500c8),name=sanity-ec.sh 00000002:00010000:0.0:1787827204.542378:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x39:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.542387:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.542397:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.542406:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.542411:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x39:0x0](ffff8ab3d1f500c8) 00000080:00010000:0.0:1787827204.542414:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x39:0x0](ffff8ab3d1f500c8) for lock 0x268e1a602cc22b8 00000080:00010000:0.0:1787827204.542419:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc22b8 from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.542422:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.542431:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.542445:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x39:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.542452:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x39:0x0] 00000008:00010000:0.0:1787827204.542463:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.542466:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f730, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.542494:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc22bf lrc: 3/1,0 mode: --/PR res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.542506:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc22bf lrc: 3/1,0 mode: --/PR res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.542546:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1787827204.543508:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x39:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787827204.543515:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.543520:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.543523:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f730 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.543528:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.543531:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f730 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.543540:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x39:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.543640:0:48442:0:(file.c:2652:do_file_write_iter()) START file sanity-ec.sh:[0x240001b71:0x39:0x0], ppos: 0, count: 123804 00000080:00200000:0.0:1787827204.543647:0:48442:0:(file.c:2090:ll_file_io_generic()) sanity-ec.sh: write ppos: 0, bytes: 123804 00000080:00200000:0.0:1787827204.543652:0:48442:0:(file.c:1848:ll_io_set_mirror()) sanity-ec.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.543658:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@00000000da0a4c62[0x110, 1, [0x240001b71:0x39:0x0] hash] 00000020:00200000:0.0:1787827204.543660:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 123804) 0 0 00000080:00200000:0.0:1787827204.543663:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x39:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.543671:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 30] 00020000:00200000:0.0:1787827204.543678:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 123804) 123804, 123804 00020000:00200000:0.0:1787827204.543682:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.543693:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 30] 00000008:00010000:0.0:1787827204.543701:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.543705:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f730, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.543713:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3c7ca6800 type 11 mode 2 res 10737419264/157 (0 126975) 00010000:00010000:0.0:1787827204.543743:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.543754:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->126975] (req 0->126975) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.543763:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->126975] (req 0->126975) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.543781:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af8780 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280544896:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.543820:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c180 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.544535:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:0.0:1787827204.544580:0:48409:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294800384 00000100:00100000:0.0:1787827204.544588:0:48409:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294800384 00000100:00100000:0.0:1787827204.544631:0:48409:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 266 00000100:00100000:0.0:1787827204.544643:0:48409:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3e0ba4780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:82330:x1874671294800384:12345-192.168.203.124@tcp:104: 00010000:00010000:0.0:1787827204.544654:0:48409:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc22bf lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da0fd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.544711:0:48409:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1787827204.544719:0:48409:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3e0ba4780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:82330:x1874671294800384:12345-192.168.203.124@tcp:104: Request processed in 79us (185us total) trans 0 rc 0/0 00000100:00100000:0.0:1787827204.544730:0:48409:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 266 00010000:00010000:3.0:1787827204.544738:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.544745:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc22bf lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da0fd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.544753:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cf400 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.544757:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc22bf lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da0fd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.544769:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x39:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.544776:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x39:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.544798:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc22bf lrc: 2/0,0 mode: --/PR res: [0x280000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da0fd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.544846:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.544867:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc22bf lrc: 1/0,0 mode: --/PR res: [0x280000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da0fd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.544877:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d622df3/0x268e1a602cc22bf lrc: 0/0,0 mode: --/PR res: [0x280000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da0fd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.544913:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.544916:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.544919:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.545456:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af8780 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280544896:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.545471:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->126975] (req 0->126975) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.545483:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->126975] (req 0->126975) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.545492:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->126975] (req 0->126975) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.545828:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.545841:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.545854:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.545864:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.545876:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.545912:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.545931:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) sanity-ec.sh: write [0, 123804) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.545944:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.545986:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:0.0:1787827204.545996:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000080204582[3 000000005132749d 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.546001:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000010e88930 17ffffc0001001 3:0 ffff8ab3d08429f8 0 lru 00000080:00200000:0.0:1787827204.546009:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000005a21a34b 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546012:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000080204582 00000080:00200000:0.0:1787827204.546014:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.546017:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546046:0:48442:0:(rw26.c:875:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546050:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000014b9c633[3 000000005132749d 2 1 000000002663f4b0 0 0 1] 00000080:00200000:0.0:1787827204.546053:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000004ecf49d8 17ffffc0001001 3:0 ffff8ab3d08420e8 1 lru 00000080:00200000:0.0:1787827204.546057:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000036bb66eb 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546059:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000014b9c633 00000080:00200000:0.0:1787827204.546060:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 2. 00000080:00200000:0.0:1787827204.546062:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546073:0:48442:0:(rw26.c:875:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546076:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000092da0e2d[3 000000005132749d 2 1 000000002663f4b0 0 0 2] 00000080:00200000:0.0:1787827204.546078:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000062dfaf14 17ffffc0001001 3:0 ffff8ab3d0842740 2 lru 00000080:00200000:0.0:1787827204.546082:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ae34047c 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546083:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000092da0e2d 00000080:00200000:0.0:1787827204.546084:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 3. 00000080:00200000:0.0:1787827204.546085:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546094:0:48442:0:(rw26.c:875:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546111:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000cc7c8c1e[3 000000005132749d 2 1 000000002663f4b0 0 0 3] 00000080:00200000:0.0:1787827204.546113:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000053bfe7d1 17ffffc0001001 3:0 ffff8ab3d0842570 3 lru 00000080:00200000:0.0:1787827204.546117:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ae45d3ff 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546119:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000cc7c8c1e 00000080:00200000:0.0:1787827204.546120:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 4. 00000080:00200000:0.0:1787827204.546121:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546133:0:48442:0:(rw26.c:875:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546136:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000073e455cb[3 000000005132749d 2 1 000000002663f4b0 0 0 4] 00000080:00200000:0.0:1787827204.546138:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000009ac33565 17ffffc0001001 3:0 ffff8ab3d08421d0 4 lru 00000080:00200000:0.0:1787827204.546142:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000fd6eaa48 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546144:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000073e455cb 00000080:00200000:0.0:1787827204.546144:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 5. 00000080:00200000:0.0:1787827204.546145:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546161:0:48442:0:(rw26.c:875:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546164:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000a31019ba[3 000000005132749d 2 1 000000002663f4b0 0 0 5] 00000080:00200000:0.0:1787827204.546166:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000de134007 17ffffc0001001 3:0 ffff8ab3d0842658 5 lru 00000080:00200000:0.0:1787827204.546169:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000226e1d4c 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546171:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000a31019ba 00000080:00200000:0.0:1787827204.546172:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 6. 00000080:00200000:0.0:1787827204.546174:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546183:0:48442:0:(rw26.c:875:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546185:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000043e76818[3 000000005132749d 2 1 000000002663f4b0 0 0 6] 00000080:00200000:0.0:1787827204.546187:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000009921b51c 17ffffc0001001 3:0 ffff8ab3d0842000 6 lru 00000080:00200000:0.0:1787827204.546191:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000003e7cff60 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546194:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000043e76818 00000080:00200000:0.0:1787827204.546195:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 7. 00000080:00200000:0.0:1787827204.546197:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546210:0:48442:0:(rw26.c:875:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546216:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000094b6f619[3 000000005132749d 2 1 000000002663f4b0 0 0 7] 00000080:00200000:0.0:1787827204.546219:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000096bc0f71 17ffffc0001001 3:0 ffff8ab3d0842e80 7 lru 00000080:00200000:0.0:1787827204.546225:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c65a3b96 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546227:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000094b6f619 00000080:00200000:0.0:1787827204.546229:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 8. 00000080:00200000:0.0:1787827204.546232:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546249:0:48442:0:(rw26.c:875:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546254:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000042c80815[3 000000005132749d 2 1 000000002663f4b0 0 0 8] 00000080:00200000:0.0:1787827204.546258:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000c8993bd4 17ffffc0001001 3:0 ffff8ab3d0842ae0 8 lru 00000080:00200000:0.0:1787827204.546262:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000009766bcac 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546264:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000042c80815 00000080:00200000:0.0:1787827204.546265:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 9. 00000080:00200000:0.0:1787827204.546267:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546277:0:48442:0:(rw26.c:875:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546288:0:48442:0:(rw26.c:933:ll_write_end()) page@000000001f329bd1[3 000000005132749d 2 1 000000002663f4b0 0 0 9] 00000080:00200000:0.0:1787827204.546290:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000cae58136 17ffffc0001001 3:0 ffff8ab3d0842bc8 9 lru 00000080:00200000:0.0:1787827204.546294:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c0e69e6c 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546297:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000001f329bd1 00000080:00200000:0.0:1787827204.546298:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 10. 00000080:00200000:0.0:1787827204.546299:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546311:0:48442:0:(rw26.c:875:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546314:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000122fe02b[3 000000005132749d 2 1 000000002663f4b0 0 0 10] 00000080:00200000:0.0:1787827204.546316:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000000c791bff 17ffffc0001001 3:0 ffff8ab3d0842910 10 lru 00000080:00200000:0.0:1787827204.546321:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000d01c7d09 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546324:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000122fe02b 00000080:00200000:0.0:1787827204.546325:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 11. 00000080:00200000:0.0:1787827204.546326:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546336:0:48442:0:(rw26.c:875:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546340:0:48442:0:(rw26.c:933:ll_write_end()) page@000000006ed3cbbe[3 000000005132749d 2 1 000000002663f4b0 0 0 11] 00000080:00200000:0.0:1787827204.546342:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000fa42cbb5 17ffffc0001001 3:0 ffff8ab3d0842828 11 lru 00000080:00200000:0.0:1787827204.546346:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000bfd7d13a 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546348:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000006ed3cbbe 00000080:00200000:0.0:1787827204.546350:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 12. 00000080:00200000:0.0:1787827204.546351:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546362:0:48442:0:(rw26.c:875:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546368:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000352b70b4[3 000000005132749d 2 1 000000002663f4b0 0 0 12] 00000080:00200000:0.0:1787827204.546371:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000e2cff53c 17ffffc0001001 3:0 ffff8ab3d0842cb0 12 lru 00000080:00200000:0.0:1787827204.546375:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000002c27f4a0 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546377:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000352b70b4 00000080:00200000:0.0:1787827204.546378:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 13. 00000080:00200000:0.0:1787827204.546380:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546389:0:48442:0:(rw26.c:875:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546392:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000c1cf61d9[3 000000005132749d 2 1 000000002663f4b0 0 0 13] 00000080:00200000:0.0:1787827204.546394:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000002a2c4527 17ffffc0001001 3:0 ffff8ab3d0842d98 13 lru 00000080:00200000:0.0:1787827204.546397:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000eae85e97 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546399:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000c1cf61d9 00000080:00200000:0.0:1787827204.546400:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 14. 00000080:00200000:0.0:1787827204.546401:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546416:0:48442:0:(rw26.c:875:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546419:0:48442:0:(rw26.c:933:ll_write_end()) page@000000004ae5e636[3 000000005132749d 2 1 000000002663f4b0 0 0 14] 00000080:00200000:0.0:1787827204.546421:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000f4230c59 17ffffc0001021 2:0 ffff8ab3d08422b8 14 lru 00000080:00200000:0.0:1787827204.546425:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000a486a3b3 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546426:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000004ae5e636 00000080:00200000:0.0:1787827204.546427:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 15. 00000080:00200000:0.0:1787827204.546428:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546438:0:48442:0:(rw26.c:875:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546441:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000da01f678[3 000000005132749d 2 1 000000002663f4b0 0 0 15] 00000080:00200000:0.0:1787827204.546443:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000002dc4d25 17ffffc0001001 3:0 ffff8ab3d0842488 15 lru 00000080:00200000:0.0:1787827204.546447:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000028f20e23 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546448:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000da01f678 00000080:00200000:0.0:1787827204.546449:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 16. 00000080:00200000:0.0:1787827204.546450:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546458:0:48442:0:(rw26.c:875:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546461:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000f31ea120[3 000000005132749d 2 1 000000002663f4b0 0 0 16] 00000080:00200000:0.0:1787827204.546462:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000abf31802 17ffffc0001001 3:0 ffff8ab3d08423a0 16 lru 00000080:00200000:0.0:1787827204.546466:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000538b808e 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546467:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000f31ea120 00000080:00200000:0.0:1787827204.546468:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 17. 00000080:00200000:0.0:1787827204.546470:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546485:0:48442:0:(rw26.c:875:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546488:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000851d5de1[3 000000005132749d 2 1 000000002663f4b0 0 0 17] 00000080:00200000:0.0:1787827204.546490:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000008f996a71 17ffffc0001001 3:0 ffff8ab3f07b3488 17 lru 00000080:00200000:0.0:1787827204.546494:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000000ecddfd 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546496:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000851d5de1 00000080:00200000:0.0:1787827204.546497:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 18. 00000080:00200000:0.0:1787827204.546498:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546506:0:48442:0:(rw26.c:875:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546509:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000ff354dd1[3 000000005132749d 2 1 000000002663f4b0 0 0 18] 00000080:00200000:0.0:1787827204.546511:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000004cb6001b 17ffffc0001001 3:0 ffff8ab3f07b33a0 18 lru 00000080:00200000:0.0:1787827204.546514:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000771ac974 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546516:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000ff354dd1 00000080:00200000:0.0:1787827204.546516:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 19. 00000080:00200000:0.0:1787827204.546518:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546526:0:48442:0:(rw26.c:875:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546528:0:48442:0:(rw26.c:933:ll_write_end()) page@000000006dae7096[3 000000005132749d 2 1 000000002663f4b0 0 0 19] 00000080:00200000:0.0:1787827204.546530:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000c4c3576c 17ffffc0001001 3:0 ffff8ab3f07b39f8 19 lru 00000080:00200000:0.0:1787827204.546534:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000006512e40f 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546535:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000006dae7096 00000080:00200000:0.0:1787827204.546536:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 20. 00000080:00200000:0.0:1787827204.546537:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546546:0:48442:0:(rw26.c:875:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546548:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000253e2ff4[3 000000005132749d 2 1 000000002663f4b0 0 0 20] 00000080:00200000:0.0:1787827204.546550:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000d3e7b0fe 17ffffc0001001 3:0 ffff8ab3f07b30e8 20 lru 00000080:00200000:0.0:1787827204.546553:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000004edd5c0 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546555:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000253e2ff4 00000080:00200000:0.0:1787827204.546555:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 21. 00000080:00200000:0.0:1787827204.546556:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546565:0:48442:0:(rw26.c:875:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546567:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000017b6d5ac[3 000000005132749d 2 1 000000002663f4b0 0 0 21] 00000080:00200000:0.0:1787827204.546569:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000007a4ca4b5 17ffffc0001001 3:0 ffff8ab3f07b3740 21 lru 00000080:00200000:0.0:1787827204.546573:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000a9fe56bb 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546574:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000017b6d5ac 00000080:00200000:0.0:1787827204.546574:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 22. 00000080:00200000:0.0:1787827204.546575:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546584:0:48442:0:(rw26.c:875:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546587:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000bcf0448f[3 000000005132749d 2 1 000000002663f4b0 0 0 22] 00000080:00200000:0.0:1787827204.546589:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000056296695 17ffffc0001001 3:0 ffff8ab3f07b3570 22 lru 00000080:00200000:0.0:1787827204.546592:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c8ed9395 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546593:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000bcf0448f 00000080:00200000:0.0:1787827204.546594:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 23. 00000080:00200000:0.0:1787827204.546595:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546603:0:48442:0:(rw26.c:875:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546605:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000e2fdeaaf[3 000000005132749d 2 1 000000002663f4b0 0 0 23] 00000080:00200000:0.0:1787827204.546607:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000006e1ebcb 17ffffc0001001 3:0 ffff8ab3f07b31d0 23 lru 00000080:00200000:0.0:1787827204.546611:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000067f487d6 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546612:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000e2fdeaaf 00000080:00200000:0.0:1787827204.546613:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 24. 00000080:00200000:0.0:1787827204.546615:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546624:0:48442:0:(rw26.c:875:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546626:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000e38ddf6b[3 000000005132749d 2 1 000000002663f4b0 0 0 24] 00000080:00200000:0.0:1787827204.546628:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000021e4b19e 17ffffc0001001 3:0 ffff8ab3f07b3658 24 lru 00000080:00200000:0.0:1787827204.546632:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000000f350434 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546634:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000e38ddf6b 00000080:00200000:0.0:1787827204.546634:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 25. 00000080:00200000:0.0:1787827204.546635:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546643:0:48442:0:(rw26.c:875:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546646:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000029d6a8e1[3 000000005132749d 2 1 000000002663f4b0 0 0 25] 00000080:00200000:0.0:1787827204.546647:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000008c7915fa 17ffffc0001001 3:0 ffff8ab3f07b3000 25 lru 00000080:00200000:0.0:1787827204.546651:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000b09a455e 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546653:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000029d6a8e1 00000080:00200000:0.0:1787827204.546653:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 26. 00000080:00200000:0.0:1787827204.546655:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546663:0:48442:0:(rw26.c:875:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546666:0:48442:0:(rw26.c:933:ll_write_end()) page@000000000199fa95[3 000000005132749d 2 1 000000002663f4b0 0 0 26] 00000080:00200000:0.0:1787827204.546668:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000053405318 17ffffc0001001 3:0 ffff8ab3f07b3e80 26 lru 00000080:00200000:0.0:1787827204.546678:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e758ed42 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546680:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000000199fa95 00000080:00200000:0.0:1787827204.546681:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 27. 00000080:00200000:0.0:1787827204.546682:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546692:0:48442:0:(rw26.c:875:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546695:0:48442:0:(rw26.c:933:ll_write_end()) page@000000008d3a0dd1[3 000000005132749d 2 1 000000002663f4b0 0 0 27] 00000080:00200000:0.0:1787827204.546697:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000007bdf49a0 17ffffc0001001 3:0 ffff8ab3f07b3ae0 27 lru 00000080:00200000:0.0:1787827204.546702:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000011638d05 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546704:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000008d3a0dd1 00000080:00200000:0.0:1787827204.546704:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 28. 00000080:00200000:0.0:1787827204.546705:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546713:0:48442:0:(rw26.c:875:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546716:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000a2cc9ce3[3 000000005132749d 2 1 000000002663f4b0 0 0 28] 00000080:00200000:0.0:1787827204.546718:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000f0167a7a 17ffffc0001001 3:0 ffff8ab3f07b3bc8 28 lru 00000080:00200000:0.0:1787827204.546722:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000b4bb6c33 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546724:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000a2cc9ce3 00000080:00200000:0.0:1787827204.546725:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 29. 00000080:00200000:0.0:1787827204.546726:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.546737:0:48442:0:(rw26.c:875:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:0.0:1787827204.546740:0:48442:0:(rw26.c:933:ll_write_end()) page@000000000fed2436[3 000000005132749d 2 1 000000002663f4b0 0 0 29] 00000080:00200000:0.0:1787827204.546741:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000030fb356e 17ffffc0001021 2:0 ffff8ab3f07b3910 29 lru 00000080:00200000:0.0:1787827204.546745:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000018aad6d4 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546747:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000000fed2436 00000080:00200000:0.0:1787827204.546748:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 30. 00000080:00200000:0.0:1787827204.546749:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 30 of 0 to 924 bytes 00000080:00200000:0.0:1787827204.546761:0:48442:0:(rw26.c:875:ll_write_end()) pos 122880, len 924, copied 924 00000080:00200000:0.0:1787827204.546764:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000c8df84f1[3 000000005132749d 2 1 000000002663f4b0 0 0 30] 00000080:00200000:0.0:1787827204.546765:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000143c5e5e 17ffffc0001001 3:0 ffff8ab3f07b3828 30 lru 00000080:00200000:0.0:1787827204.546769:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e08e7427 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 00000000f5856a27 00000000240f5fd6 > 3< 0 0 > 4< 0 0 8 2826240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.546771:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000c8df84f1 00000080:00200000:0.0:1787827204.546772:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 31. 00000080:00200000:0.0:1787827204.546775:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 31, from 0, to 924 prio 0 00000008:00010000:0.0:1787827204.546790:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.546805:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.546816:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.546827:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.546843:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.546853:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.546863:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.546873:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.546882:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.546918:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.546929:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.546939:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.546949:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.546959:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.546969:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.546986:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f50278, count 15, dirtied 15 00000008:00010000:0.0:1787827204.546994:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.547003:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.547013:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.547023:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.547032:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.547042:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.547059:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.547070:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.547079:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.547089:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.547117:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.547127:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.547138:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.547148:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.547159:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.547173:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f50278, count 15, dirtied 15 00000008:00010000:0.0:1787827204.547180:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.547190:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f50278, count 1, dirtied 1 00000080:00200000:0.0:1787827204.547193:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 31 pages 123804 bytes, tot: 123804 00000080:00200000:0.0:1787827204.547196:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) sanity-ec.sh: write bytes 0, result: 123804 00000020:00010000:0.0:1787827204.547211:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.547215:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.547218:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f730 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.547223:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.547233:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.547244:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.547248:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f730 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.547254:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 123804 bytes 00000080:00200000:0.0:1787827204.547259:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 30] 00000080:00200000:0.0:1787827204.547265:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x39:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.547270:0:48442:0:(file.c:2240:ll_file_io_generic()) sanity-ec.sh: 2 io complete with rc: 0, result: 123804, restart: 0 00000080:00200000:0.0:1787827204.547276:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x39:0x0] updating i_size 123804 i_blocks 0 00000080:00200000:0.0:1787827204.547280:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 123804 00000080:00200008:0.0:1787827204.547304:0:48442:0:(file.c:479:ll_file_release()) START file sanity-ec.sh:[0x240001b71:0x39:0x0](ffff8ab3d1f500c8), flags 100001 00000002:00100000:0.0:1787827204.547345:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3d0afb840 x1874671280544640/t17179869486(17179869486) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.547365:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.547371:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af8780 x1874671280545280/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.547388:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af8780 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280545280:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.547427:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c180 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.548286:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af8780 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280545280:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.548304:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3d0afb840 x1874671280544640/t17179869486(17179869486) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.548349:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.548362:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.548374:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.548386:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.548395:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.548400:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.548404:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.548410:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.548415:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.548424:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.548442:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.548449:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.548455:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.548459:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.548463:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.548467:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-ec.sh, flags=8 00000080:00200008:0.0:1787827204.548474:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-ec.sh:[0x240001b71:0x39:0x0](ffff8ab3d1f500c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.548479:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x39:0x0](ffff8ab3d1f500c8),name=sanity-ec.sh 00000002:00010000:0.0:1787827204.548490:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x39:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.548499:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.548510:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.548518:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.548523:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x39:0x0](ffff8ab3d1f500c8) 00000080:00010000:0.0:1787827204.548526:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x39:0x0](ffff8ab3d1f500c8) for lock 0x268e1a602cc22b8 00000080:00010000:0.0:1787827204.548531:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc22b8 from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.548535:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.548544:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.548558:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x39:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.548565:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x39:0x0] 00000008:00010000:0.0:1787827204.548574:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.548578:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f730, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.548589:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.548601:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.548613:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.548617:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.548627:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.548639:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.548644:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x39:0x0] updating i_size 123804 i_blocks 0 00000020:00010000:0.0:1787827204.548648:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.548651:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.548654:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f730 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.548658:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.548660:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f730 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.548667:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x39:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.548708:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.548718:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.548727:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.548739:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.548747:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.548752:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.548755:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.548759:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.548763:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.548772:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.548783:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.548788:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.548792:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.548796:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.548799:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.548810:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-flr.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.548815:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-flr.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.548826:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-flr.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.548855:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc22cd lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.548864:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc22cd lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.548879:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc22cd lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.548927:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af9a40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280545408:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.548969:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c180 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.549872:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af9a40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280545408:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.549913:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc22cd lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.549925:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc22cd lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.549932:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc22cd lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.549940:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc22cd lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.549947:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc22cd lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.549954:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cea00 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.549958:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc22cd lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.549967:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc22cd lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.549975:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc22cd lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.549983:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.549989:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc22cd lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.549997:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc22cd lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.550012:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0af9a40 x1874671280545408/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.550024:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.550038:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.550067:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.550075:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.550083:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.550094:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.550119:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.550123:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.550127:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.550131:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.550135:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.550145:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.550156:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.550161:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.550165:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.550169:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.550172:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.550181:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-flr.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.550185:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-flr.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.550193:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-flr.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.550215:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc22d4 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.550223:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc22d4 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.550232:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc22d4 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.550246:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af9a40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280545536:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.550283:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c180 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.551247:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af9a40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280545536:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.551259:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc22d4 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.551268:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc22d4 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.551275:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc22d4 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.551283:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc22d4 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.551290:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc22d4 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.551296:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cf600 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.551300:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc22d4 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.551307:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc22d4 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.551314:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc22d4 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.551319:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.551323:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc22d4 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.551331:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc22d4 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.551343:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0af9a40 x1874671280545536/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.551354:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.551365:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.551419:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.551430:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.551438:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.551450:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.551458:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.551462:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.551465:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.551470:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.551474:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.551483:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.551493:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.551499:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.551503:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.551507:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.551510:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.551521:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.551526:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=sanity-flr.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be3b00, open_flags 80c1, mode 81ed opened 0 00000080:00200000:0.0:1787827204.551534:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-flr.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.551549:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-flr.sh,[0x240001b71:0x3a:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.551582:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.551590:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.551600:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.551608:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af9a40 x1874671280545792/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.551618:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.551633:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af9a40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280545792:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.551734:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c180 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.553051:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af9a40 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280545792:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.553066:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.553084:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.553094:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.553120:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.553131:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.553143:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.553158:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.553168:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.553179:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3d0af9a40 x1874671280545792/t17179869488(17179869488) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.553193:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0af9a40 x1874671280545792/t17179869488(17179869488) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.553207:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.553221:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.553231:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/58 (0 0) 00000080:00200000:0.0:1787827204.553236:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.553244:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=sanity-flr.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.553268:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x3a:0x0](ffff8ab3d1f54b08) 00000080:00200000:0.0:1787827204.553348:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x3a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.553354:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x3a:0x0](ffff8ab3d1f54b08) 00000080:00010000:0.0:1787827204.553358:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x3a:0x0](ffff8ab3d1f54b08) for lock 0x268e1a602cc22db 00000080:00010000:0.0:1787827204.553367:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x3a:0x0](ffff8ab3d1f54b08) 00000080:00200008:0.0:1787827204.553375:0:48442:0:(file.c:1090:ll_file_open()) START file sanity-flr.sh:[0x240001b71:0x3a:0x0](ffff8ab3d1f54b08), flags 100301 00000002:00100000:0.0:1787827204.553394:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0af9a40 x1874671280545792/t17179869488(17179869488) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.553429:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc22db from it ffff8ab3d0a44f60 00010000:00010000:0.0:1787827204.553435:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.553445:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.553461:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x3a:0x0](ffff8ab3d1f54b08), inode mode 81ed mask 40 00000080:00200008:0.0:1787827204.553474:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-flr.sh:[0x240001b71:0x3a:0x0](ffff8ab3d1f54b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.553479:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x3a:0x0](ffff8ab3d1f54b08),name=sanity-flr.sh 00000002:00010000:0.0:1787827204.553490:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x3a:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.553497:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.553507:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.553515:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.553520:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x3a:0x0](ffff8ab3d1f54b08) 00000080:00010000:0.0:1787827204.553523:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x3a:0x0](ffff8ab3d1f54b08) for lock 0x268e1a602cc22db 00000080:00010000:0.0:1787827204.553527:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc22db from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.553531:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.553541:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.553555:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3a:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.553562:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x3a:0x0] 00000008:00010000:0.0:1787827204.553572:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.553575:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f230, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.553602:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc22e2 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.553613:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc22e2 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.553643:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1787827204.554654:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x3a:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787827204.554663:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.554681:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.554688:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f230 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.554693:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.554697:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f230 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.554707:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3a:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.554813:0:48442:0:(file.c:2652:do_file_write_iter()) START file sanity-flr.sh:[0x240001b71:0x3a:0x0], ppos: 0, count: 145387 00000080:00200000:0.0:1787827204.554822:0:48442:0:(file.c:2090:ll_file_io_generic()) sanity-flr.sh: write ppos: 0, bytes: 145387 00000080:00200000:0.0:1787827204.554828:0:48442:0:(file.c:1848:ll_io_set_mirror()) sanity-flr.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.554835:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@00000000a9e3078c[0x110, 1, [0x240001b71:0x3a:0x0] hash] 00000020:00200000:0.0:1787827204.554837:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 145387) 0 0 00000080:00200000:0.0:1787827204.554840:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3a:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.554848:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 35] 00020000:00200000:0.0:1787827204.554856:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 145387) 145387, 145387 00020000:00200000:0.0:1787827204.554860:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.554872:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 35] 00000008:00010000:0.0:1787827204.554882:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.554906:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f230, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.554915:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d2d3cc00 type 11 mode 2 res 11811161088/158 (0 147455) 00010000:00010000:0.0:1787827204.554939:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.554950:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->147455] (req 0->147455) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.554961:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->147455] (req 0->147455) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.554979:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afa580 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280546048:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.555021:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c840 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.555794:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:0.0:1787827204.555841:0:48409:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294800640 00000100:00100000:0.0:1787827204.555848:0:48409:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294800640 00010000:00010000:3.0:1787827204.556026:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.556034:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc22e2 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da158 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.556043:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07ce400 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.556047:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc22e2 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da158 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.556060:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.556066:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.556091:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc22e2 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da158 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.556144:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.556165:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc22e2 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da158 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.556176:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc22e2 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da158 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.556189:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.556191:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.556195:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.556616:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afa580 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280546048:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.556630:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->147455] (req 0->147455) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.556642:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->147455] (req 0->147455) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.556654:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->147455] (req 0->147455) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.557069:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:0.0:1787827204.557243:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.557262:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.557283:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.557296:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.557309:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.557319:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.557338:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) sanity-flr.sh: write [0, 145387) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.557353:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557392:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557403:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000ba696540[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.557407:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000001ffd84b7 17ffffc0001001 3:0 ffff8ab3f07b3cb0 0 lru 00000080:00200000:0.0:1787827204.557415:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000085517031 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557418:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000ba696540 00000080:00200000:0.0:1787827204.557419:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.557423:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557454:0:48442:0:(rw26.c:875:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557464:0:48442:0:(rw26.c:933:ll_write_end()) page@000000005c0ab58b[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 1] 00000080:00200000:0.0:1787827204.557467:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000169989a5 17ffffc0001001 3:0 ffff8ab3f07b3d98 1 lru 00000080:00200000:0.0:1787827204.557473:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000008f7f15f8 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557476:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000005c0ab58b 00000080:00200000:0.0:1787827204.557477:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 2. 00000080:00200000:0.0:1787827204.557479:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557494:0:48442:0:(rw26.c:875:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557499:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000ebcaca47[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 2] 00000080:00200000:0.0:1787827204.557502:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000056357dca 17ffffc0001001 3:0 ffff8ab3f07b32b8 2 lru 00000080:00200000:0.0:1787827204.557508:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000f919e0dc 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557510:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000ebcaca47 00000080:00200000:0.0:1787827204.557512:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 3. 00000080:00200000:0.0:1787827204.557514:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557539:0:48442:0:(rw26.c:875:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557544:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000b8e5ce01[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 3] 00000080:00200000:0.0:1787827204.557548:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000126273a9 17ffffc0001001 3:0 ffff8ab3d0eb4e80 3 lru 00000080:00200000:0.0:1787827204.557554:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000f0091e4a 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557557:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000b8e5ce01 00000080:00200000:0.0:1787827204.557558:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 4. 00000080:00200000:0.0:1787827204.557560:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557574:0:48442:0:(rw26.c:875:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557580:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000009a3eecb[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 4] 00000080:00200000:0.0:1787827204.557583:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000f3ea8b33 17ffffc0001001 3:0 ffff8ab3d0eb4ae0 4 lru 00000080:00200000:0.0:1787827204.557589:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000a39db692 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557591:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000009a3eecb 00000080:00200000:0.0:1787827204.557593:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 5. 00000080:00200000:0.0:1787827204.557595:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557613:0:48442:0:(rw26.c:875:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557619:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000307b628b[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 5] 00000080:00200000:0.0:1787827204.557622:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000000d14d87f 17ffffc0001001 3:0 ffff8ab3d0eb4bc8 5 lru 00000080:00200000:0.0:1787827204.557628:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000021029149 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557631:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000307b628b 00000080:00200000:0.0:1787827204.557632:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 6. 00000080:00200000:0.0:1787827204.557635:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557652:0:48442:0:(rw26.c:875:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557655:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000634ad693[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 6] 00000080:00200000:0.0:1787827204.557658:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000044396c1d 17ffffc0001001 3:0 ffff8ab3d0eb4910 6 lru 00000080:00200000:0.0:1787827204.557661:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000686e8d60 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557663:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000634ad693 00000080:00200000:0.0:1787827204.557664:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 7. 00000080:00200000:0.0:1787827204.557665:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557675:0:48442:0:(rw26.c:875:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557678:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000028c4aeec[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 7] 00000080:00200000:0.0:1787827204.557680:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000005c239c5f 17ffffc0001001 3:0 ffff8ab3d0eb4828 7 lru 00000080:00200000:0.0:1787827204.557684:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000005b660143 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557685:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000028c4aeec 00000080:00200000:0.0:1787827204.557686:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 8. 00000080:00200000:0.0:1787827204.557687:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557696:0:48442:0:(rw26.c:875:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557698:0:48442:0:(rw26.c:933:ll_write_end()) page@000000006bab9bb5[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 8] 00000080:00200000:0.0:1787827204.557700:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000072c904b5 17ffffc0001001 3:0 ffff8ab3d0eb4cb0 8 lru 00000080:00200000:0.0:1787827204.557704:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000a818dca6 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557706:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000006bab9bb5 00000080:00200000:0.0:1787827204.557707:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 9. 00000080:00200000:0.0:1787827204.557708:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557716:0:48442:0:(rw26.c:875:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557719:0:48442:0:(rw26.c:933:ll_write_end()) page@000000007d6e5056[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 9] 00000080:00200000:0.0:1787827204.557721:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000b258dab5 17ffffc0001001 3:0 ffff8ab3d0eb4d98 9 lru 00000080:00200000:0.0:1787827204.557724:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000cde85892 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557726:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000007d6e5056 00000080:00200000:0.0:1787827204.557727:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 10. 00000080:00200000:0.0:1787827204.557728:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557737:0:48442:0:(rw26.c:875:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557739:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000537b5830[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 10] 00000080:00200000:0.0:1787827204.557741:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000f682a656 17ffffc0001001 3:0 ffff8ab3d0eb42b8 10 lru 00000080:00200000:0.0:1787827204.557745:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000005d4aaac9 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557747:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000537b5830 00000080:00200000:0.0:1787827204.557748:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 11. 00000080:00200000:0.0:1787827204.557749:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557758:0:48442:0:(rw26.c:875:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557760:0:48442:0:(rw26.c:933:ll_write_end()) page@000000003beeb1db[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 11] 00000080:00200000:0.0:1787827204.557762:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000000afd7a4b 17ffffc0001001 3:0 ffff8ab3d0eb4488 11 lru 00000080:00200000:0.0:1787827204.557766:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000243736e1 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557768:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000003beeb1db 00000080:00200000:0.0:1787827204.557769:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 12. 00000080:00200000:0.0:1787827204.557770:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557781:0:48442:0:(rw26.c:875:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557784:0:48442:0:(rw26.c:933:ll_write_end()) page@000000008f13610d[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 12] 00000080:00200000:0.0:1787827204.557786:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000eda897c5 17ffffc0001001 3:0 ffff8ab3d0eb43a0 12 lru 00000080:00200000:0.0:1787827204.557790:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000bc5b133d 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557792:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000008f13610d 00000080:00200000:0.0:1787827204.557793:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 13. 00000080:00200000:0.0:1787827204.557794:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557809:0:48442:0:(rw26.c:875:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557812:0:48442:0:(rw26.c:933:ll_write_end()) page@000000007dde09d8[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 13] 00000080:00200000:0.0:1787827204.557814:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000026978fd9 17ffffc0001021 2:0 ffff8ab3d0eb49f8 13 lru 00000080:00200000:0.0:1787827204.557817:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000006d48e6ec 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557820:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000007dde09d8 00000080:00200000:0.0:1787827204.557821:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 14. 00000080:00200000:0.0:1787827204.557822:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557831:0:48442:0:(rw26.c:875:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557835:0:48442:0:(rw26.c:933:ll_write_end()) page@000000009359ce30[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 14] 00000080:00200000:0.0:1787827204.557837:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000097591aa2 17ffffc0001001 3:0 ffff8ab3d0eb40e8 14 lru 00000080:00200000:0.0:1787827204.557841:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000027fed4b 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557843:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000009359ce30 00000080:00200000:0.0:1787827204.557844:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 15. 00000080:00200000:0.0:1787827204.557846:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557855:0:48442:0:(rw26.c:875:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557858:0:48442:0:(rw26.c:933:ll_write_end()) page@000000006767867d[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 15] 00000080:00200000:0.0:1787827204.557861:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000038d0279a 17ffffc0001001 3:0 ffff8ab3d0eb4740 15 lru 00000080:00200000:0.0:1787827204.557865:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000075611063 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557868:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000006767867d 00000080:00200000:0.0:1787827204.557869:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 16. 00000080:00200000:0.0:1787827204.557870:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557880:0:48442:0:(rw26.c:875:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557901:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000c6c20afc[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 16] 00000080:00200000:0.0:1787827204.557905:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000059ce6195 17ffffc0001001 3:0 ffff8ab3d0eb4570 16 lru 00000080:00200000:0.0:1787827204.557909:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000006a800ffc 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557912:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000c6c20afc 00000080:00200000:0.0:1787827204.557913:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 17. 00000080:00200000:0.0:1787827204.557916:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557928:0:48442:0:(rw26.c:875:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557932:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000cd7197c9[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 17] 00000080:00200000:0.0:1787827204.557935:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000000383c112 17ffffc0001001 3:0 ffff8ab3d0eb41d0 17 lru 00000080:00200000:0.0:1787827204.557939:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000f186b25a 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557942:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000cd7197c9 00000080:00200000:0.0:1787827204.557943:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 18. 00000080:00200000:0.0:1787827204.557945:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557955:0:48442:0:(rw26.c:875:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557960:0:48442:0:(rw26.c:933:ll_write_end()) page@000000008144850b[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 18] 00000080:00200000:0.0:1787827204.557964:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000de9274a7 17ffffc0001001 3:0 ffff8ab3d0eb4658 18 lru 00000080:00200000:0.0:1787827204.557972:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000000f2053e6 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.557974:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000008144850b 00000080:00200000:0.0:1787827204.557975:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 19. 00000080:00200000:0.0:1787827204.557976:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.557990:0:48442:0:(rw26.c:875:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:0.0:1787827204.557993:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000af63ad3d[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 19] 00000080:00200000:0.0:1787827204.557995:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000009542fa22 17ffffc0001001 3:0 ffff8ab3d0eb4000 19 lru 00000080:00200000:0.0:1787827204.557999:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000032123da 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558001:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000af63ad3d 00000080:00200000:0.0:1787827204.558002:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 20. 00000080:00200000:0.0:1787827204.558003:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558020:0:48442:0:(rw26.c:875:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558023:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000031c24683[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 20] 00000080:00200000:0.0:1787827204.558025:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000065ae3a26 17ffffc0001001 3:0 ffff8ab3d074c488 20 lru 00000080:00200000:0.0:1787827204.558029:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000056b14b18 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558030:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000031c24683 00000080:00200000:0.0:1787827204.558031:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 21. 00000080:00200000:0.0:1787827204.558033:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558042:0:48442:0:(rw26.c:875:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558045:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000852eab88[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 21] 00000080:00200000:0.0:1787827204.558046:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000003ae6c3ce 17ffffc0001001 3:0 ffff8ab3d074c3a0 21 lru 00000080:00200000:0.0:1787827204.558050:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000000156019e 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558052:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000852eab88 00000080:00200000:0.0:1787827204.558053:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 22. 00000080:00200000:0.0:1787827204.558054:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558062:0:48442:0:(rw26.c:875:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558065:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000005b5c91d[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 22] 00000080:00200000:0.0:1787827204.558067:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000cd405a00 17ffffc0001001 3:0 ffff8ab3d074c9f8 22 lru 00000080:00200000:0.0:1787827204.558070:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000fdc44439 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558072:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000005b5c91d 00000080:00200000:0.0:1787827204.558073:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 23. 00000080:00200000:0.0:1787827204.558074:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558082:0:48442:0:(rw26.c:875:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558085:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000070ad6a03[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 23] 00000080:00200000:0.0:1787827204.558087:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000c248179c 17ffffc0001001 3:0 ffff8ab3d074c0e8 23 lru 00000080:00200000:0.0:1787827204.558091:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000008acbb747 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558092:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000070ad6a03 00000080:00200000:0.0:1787827204.558093:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 24. 00000080:00200000:0.0:1787827204.558110:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558123:0:48442:0:(rw26.c:875:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558126:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000082dc4ea3[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 24] 00000080:00200000:0.0:1787827204.558128:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000dd83b72f 17ffffc0001001 3:0 ffff8ab3d074c740 24 lru 00000080:00200000:0.0:1787827204.558132:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000acd05c45 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558134:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000082dc4ea3 00000080:00200000:0.0:1787827204.558135:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 25. 00000080:00200000:0.0:1787827204.558136:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558145:0:48442:0:(rw26.c:875:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558148:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000deedc401[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 25] 00000080:00200000:0.0:1787827204.558150:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000007b618b91 17ffffc0001001 3:0 ffff8ab3d074c570 25 lru 00000080:00200000:0.0:1787827204.558153:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000001a6a08ea 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558155:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000deedc401 00000080:00200000:0.0:1787827204.558156:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 26. 00000080:00200000:0.0:1787827204.558157:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558167:0:48442:0:(rw26.c:875:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558170:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000781e3487[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 26] 00000080:00200000:0.0:1787827204.558172:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000004f803eba 17ffffc0001001 3:0 ffff8ab3d074c1d0 26 lru 00000080:00200000:0.0:1787827204.558175:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e1ddc321 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558177:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000781e3487 00000080:00200000:0.0:1787827204.558178:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 27. 00000080:00200000:0.0:1787827204.558179:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558187:0:48442:0:(rw26.c:875:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558189:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000063a088bf[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 27] 00000080:00200000:0.0:1787827204.558191:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000003992382e 17ffffc0001001 3:0 ffff8ab3d074c658 27 lru 00000080:00200000:0.0:1787827204.558195:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c05edf99 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558196:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000063a088bf 00000080:00200000:0.0:1787827204.558197:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 28. 00000080:00200000:0.0:1787827204.558198:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558209:0:48442:0:(rw26.c:875:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558211:0:48442:0:(rw26.c:933:ll_write_end()) page@000000009165f41a[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 28] 00000080:00200000:0.0:1787827204.558213:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000c1f66fe9 17ffffc0001021 2:0 ffff8ab3d074c000 28 lru 00000080:00200000:0.0:1787827204.558218:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000005b2156ec 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558220:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000009165f41a 00000080:00200000:0.0:1787827204.558221:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 29. 00000080:00200000:0.0:1787827204.558222:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558231:0:48442:0:(rw26.c:875:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558233:0:48442:0:(rw26.c:933:ll_write_end()) page@000000006024c30c[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 29] 00000080:00200000:0.0:1787827204.558235:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000b8832d9c 17ffffc0001001 3:0 ffff8ab3d074ce80 29 lru 00000080:00200000:0.0:1787827204.558239:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000d6acbdbc 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558240:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000006024c30c 00000080:00200000:0.0:1787827204.558241:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 30. 00000080:00200000:0.0:1787827204.558243:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558252:0:48442:0:(rw26.c:875:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558254:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000330c72be[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 30] 00000080:00200000:0.0:1787827204.558256:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000005c8f6c4d 17ffffc0001001 3:0 ffff8ab3d074cae0 30 lru 00000080:00200000:0.0:1787827204.558260:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000a4873c13 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558261:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000330c72be 00000080:00200000:0.0:1787827204.558262:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 31. 00000080:00200000:0.0:1787827204.558263:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558271:0:48442:0:(rw26.c:875:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558274:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000058df71ba[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 31] 00000080:00200000:0.0:1787827204.558275:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000547e330b 17ffffc0001001 3:0 ffff8ab3d074cbc8 31 lru 00000080:00200000:0.0:1787827204.558279:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e80bef6b 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558280:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000058df71ba 00000080:00200000:0.0:1787827204.558281:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 32. 00000080:00200000:0.0:1787827204.558282:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558290:0:48442:0:(rw26.c:875:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558294:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000c6924789[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 32] 00000080:00200000:0.0:1787827204.558297:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000066bf03b7 17ffffc0001001 3:0 ffff8ab3d074c910 32 lru 00000080:00200000:0.0:1787827204.558304:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000009ccd6197 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558307:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000c6924789 00000080:00200000:0.0:1787827204.558309:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 33. 00000080:00200000:0.0:1787827204.558310:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558328:0:48442:0:(rw26.c:875:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558333:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000039b041b5[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 33] 00000080:00200000:0.0:1787827204.558336:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000005508c350 17ffffc0001001 3:0 ffff8ab3d074c828 33 lru 00000080:00200000:0.0:1787827204.558342:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000076806002 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558344:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000039b041b5 00000080:00200000:0.0:1787827204.558346:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 34. 00000080:00200000:0.0:1787827204.558348:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 34 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.558364:0:48442:0:(rw26.c:875:ll_write_end()) pos 139264, len 4096, copied 4096 00000080:00200000:0.0:1787827204.558368:0:48442:0:(rw26.c:933:ll_write_end()) page@000000007f125a16[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 34] 00000080:00200000:0.0:1787827204.558371:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000fbbb1d9e 17ffffc0001001 3:0 ffff8ab3d074ccb0 34 lru 00000080:00200000:0.0:1787827204.558377:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000f785487b 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558379:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000007f125a16 00000080:00200000:0.0:1787827204.558381:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 35. 00000080:00200000:0.0:1787827204.558383:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 35 of 0 to 2027 bytes 00000080:00200000:0.0:1787827204.558401:0:48442:0:(rw26.c:875:ll_write_end()) pos 143360, len 2027, copied 2027 00000080:00200000:0.0:1787827204.558405:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000b968c389[3 000000001ffd95fd 2 1 000000002663f4b0 0 0 35] 00000080:00200000:0.0:1787827204.558409:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000004c5460ef 17ffffc0001001 3:0 ffff8ab3d074cd98 35 lru 00000080:00200000:0.0:1787827204.558414:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e7bcd7f2 35: 1< 0 - - > 2< 143360 0 0 0x0 0x0 | 000000006a588f83 00000000e2eee58d > 3< 0 0 > 4< 0 0 8 3497984 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.558416:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000b968c389 00000080:00200000:0.0:1787827204.558418:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 36. 00000080:00200000:0.0:1787827204.558421:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 36, from 0, to 2027 prio 0 00000008:00010000:0.0:1787827204.558440:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558459:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558468:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558484:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558493:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558502:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558511:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558519:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558528:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558537:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558545:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558554:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558563:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558573:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558581:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.558598:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f54cb8, count 15, dirtied 15 00000008:00010000:0.0:1787827204.558605:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558614:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558623:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558632:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558640:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558655:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558663:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558672:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558681:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558689:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558698:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558706:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558715:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558724:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558733:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.558745:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f54cb8, count 15, dirtied 15 00000008:00010000:0.0:1787827204.558751:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558760:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558768:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558777:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558787:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.558796:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000050251337 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.558807:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3d1f54cb8, count 6, dirtied 6 00000080:00200000:0.0:1787827204.558811:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 36 pages 145387 bytes, tot: 145387 00000080:00200000:0.0:1787827204.558815:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) sanity-flr.sh: write bytes 0, result: 145387 00000020:00010000:0.0:1787827204.558829:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.558834:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.558837:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f230 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.558843:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.558854:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.558866:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.558870:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f230 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.558877:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 145387 bytes 00000080:00200000:0.0:1787827204.558908:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 35] 00000080:00200000:0.0:1787827204.558916:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3a:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.558921:0:48442:0:(file.c:2240:ll_file_io_generic()) sanity-flr.sh: 2 io complete with rc: 0, result: 145387, restart: 0 00000080:00200000:0.0:1787827204.558927:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x3a:0x0] updating i_size 145387 i_blocks 0 00000080:00200000:0.0:1787827204.558931:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 145387 00000080:00200008:0.0:1787827204.558956:0:48442:0:(file.c:479:ll_file_release()) START file sanity-flr.sh:[0x240001b71:0x3a:0x0](ffff8ab3d1f54b08), flags 100001 00000002:00100000:0.0:1787827204.558996:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3d0af9a40 x1874671280545792/t17179869488(17179869488) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.559008:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.559014:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afa580 x1874671280546432/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.559032:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afa580 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280546432:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.559081:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c840 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.560124:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afa580 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280546432:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.560146:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3d0af9a40 x1874671280545792/t17179869488(17179869488) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.560210:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.560225:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.560241:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.560254:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.560262:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.560267:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.560272:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.560277:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.560289:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.560299:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.560310:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.560319:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.560325:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.560329:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.560333:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.560336:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-flr.sh, flags=8 00000080:00200008:0.0:1787827204.560343:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-flr.sh:[0x240001b71:0x3a:0x0](ffff8ab3d1f54b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.560348:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x3a:0x0](ffff8ab3d1f54b08),name=sanity-flr.sh 00000002:00010000:0.0:1787827204.560359:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x3a:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.560368:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.560378:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.560385:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.560390:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x3a:0x0](ffff8ab3d1f54b08) 00000080:00010000:0.0:1787827204.560392:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x3a:0x0](ffff8ab3d1f54b08) for lock 0x268e1a602cc22db 00000080:00010000:0.0:1787827204.560397:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc22db from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.560400:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.560409:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.560425:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3a:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.560434:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x3a:0x0] 00000008:00010000:0.0:1787827204.560445:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.560447:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f230, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.560459:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.560471:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.560482:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.560486:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.560496:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000adc4f3ad/0x268e1a602cc22e9 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->147455) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da15f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.560507:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.560513:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x3a:0x0] updating i_size 145387 i_blocks 0 00000020:00010000:0.0:1787827204.560518:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.560522:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.560525:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f230 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.560528:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.560531:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f230 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.560538:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3a:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.560588:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.560597:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.560605:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.560615:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.560622:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.560627:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.560630:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.560634:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.560638:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.560653:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.560664:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.560668:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.560684:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.560688:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.560691:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.560705:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-hsm.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.560710:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-hsm.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.560720:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-hsm.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.560751:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf000/0x268e1a602cc22f0 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.560760:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf000/0x268e1a602cc22f0 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.560769:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf000/0x268e1a602cc22f0 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.560786:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af92c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280546560:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.560831:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c840 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.561925:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af92c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280546560:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.561938:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf000/0x268e1a602cc22f0 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.561948:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf000/0x268e1a602cc22f0 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.561956:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf000/0x268e1a602cc22f0 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.561963:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf000/0x268e1a602cc22f0 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.561972:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf000/0x268e1a602cc22f0 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.561979:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cf000 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.561984:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf000/0x268e1a602cc22f0 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.561993:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf000/0x268e1a602cc22f0 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.562001:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf000/0x268e1a602cc22f0 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.562008:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.562014:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf000/0x268e1a602cc22f0 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.562021:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf000/0x268e1a602cc22f0 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.562037:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0af92c0 x1874671280546560/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.562049:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.562064:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.562110:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.562120:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.562129:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.562139:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.562147:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.562152:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.562155:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.562160:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.562163:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.562173:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.562182:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.562188:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.562192:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.562195:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.562198:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.562207:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-hsm.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.562211:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-hsm.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.562220:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-hsm.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.562243:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc22f7 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.562256:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc22f7 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.562264:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc22f7 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.562279:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af92c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280546688:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.562320:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c840 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.563342:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af92c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280546688:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.563353:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc22f7 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.563362:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc22f7 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.563370:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc22f7 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.563376:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc22f7 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.563384:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc22f7 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.563391:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cf400 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.563395:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc22f7 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.563402:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc22f7 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.563408:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc22f7 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.563415:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.563419:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc22f7 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.563427:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc22f7 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.563440:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0af92c0 x1874671280546688/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.563451:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.563463:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.563519:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.563529:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.563537:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.563548:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.563556:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.563560:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.563563:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.563569:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.563572:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.563587:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.563598:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.563603:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.563607:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.563611:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.563614:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.563627:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.563632:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=sanity-hsm.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be3080, open_flags 80c1, mode 81ed opened 0 00000080:00200000:0.0:1787827204.563641:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-hsm.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.563657:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-hsm.sh,[0x240001b71:0x3b:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.563691:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.563699:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.563710:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.563717:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af92c0 x1874671280546944/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.563728:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.563744:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af92c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280546944:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.563857:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c840 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.565273:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af92c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280546944:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.565284:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.565297:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.565303:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.565311:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.565317:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.565323:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.565328:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.565333:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.565340:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3d0af92c0 x1874671280546944/t17179869490(17179869490) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.565348:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0af92c0 x1874671280546944/t17179869490(17179869490) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.565359:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.565368:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.565374:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/59 (0 0) 00000080:00200000:0.0:1787827204.565378:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.565384:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=sanity-hsm.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.565422:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x3b:0x0](ffff8ab3f1239988) 00000080:00200000:0.0:1787827204.565504:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x3b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.565508:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x3b:0x0](ffff8ab3f1239988) 00000080:00010000:0.0:1787827204.565511:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x3b:0x0](ffff8ab3f1239988) for lock 0x268e1a602cc22fe 00000080:00010000:0.0:1787827204.565519:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x3b:0x0](ffff8ab3f1239988) 00000080:00200008:0.0:1787827204.565526:0:48442:0:(file.c:1090:ll_file_open()) START file sanity-hsm.sh:[0x240001b71:0x3b:0x0](ffff8ab3f1239988), flags 100301 00000002:00100000:0.0:1787827204.565541:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0af92c0 x1874671280546944/t17179869490(17179869490) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.565572:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc22fe from it ffff8ab3c7b1c7e0 00010000:00010000:0.0:1787827204.565577:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.565587:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.565608:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x3b:0x0](ffff8ab3f1239988), inode mode 81ed mask 40 00000080:00200008:0.0:1787827204.565624:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-hsm.sh:[0x240001b71:0x3b:0x0](ffff8ab3f1239988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.565631:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x3b:0x0](ffff8ab3f1239988),name=sanity-hsm.sh 00000002:00010000:0.0:1787827204.565643:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x3b:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.565652:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.565662:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.565670:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.565674:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x3b:0x0](ffff8ab3f1239988) 00000080:00010000:0.0:1787827204.565677:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x3b:0x0](ffff8ab3f1239988) for lock 0x268e1a602cc22fe 00000080:00010000:0.0:1787827204.565682:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc22fe from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.565685:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.565694:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.565710:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3b:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.565720:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x3b:0x0] 00000008:00010000:0.0:1787827204.565734:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.565738:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f530, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.565772:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2305 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.565784:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2305 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.565829:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1787827204.566815:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x3b:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787827204.566822:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.566829:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.566835:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f530 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.566840:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.566844:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f530 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.566856:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3b:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.567059:0:48442:0:(file.c:2652:do_file_write_iter()) START file sanity-hsm.sh:[0x240001b71:0x3b:0x0], ppos: 0, count: 170595 00000080:00200000:0.0:1787827204.567071:0:48442:0:(file.c:2090:ll_file_io_generic()) sanity-hsm.sh: write ppos: 0, bytes: 170595 00000080:00200000:0.0:1787827204.567078:0:48442:0:(file.c:1848:ll_io_set_mirror()) sanity-hsm.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.567084:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@000000005dffd830[0x110, 1, [0x240001b71:0x3b:0x0] hash] 00000020:00200000:0.0:1787827204.567087:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 170595) 0 0 00000080:00200000:0.0:1787827204.567091:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3b:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.567115:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 41] 00020000:00200000:0.0:1787827204.567125:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 170595) 170595, 170595 00020000:00200000:0.0:1787827204.567130:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.567143:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 41] 00000008:00010000:0.0:1787827204.567155:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.567159:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f530, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.567169:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3c7ca6800 type 11 mode 2 res 10737419264/158 (0 172031) 00010000:00010000:0.0:1787827204.567191:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.567201:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->172031] (req 0->172031) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.567212:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->172031] (req 0->172031) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.567229:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af8f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280547200:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.567269:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c600 going to sleep for 11 seconds 00010000:00010000:3.0:1787827204.568387:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.568399:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2305 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da1b3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.568409:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6959e00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.568415:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2305 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da1b3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.568428:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.568436:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.568463:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2305 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da1b3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.568469:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.568544:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2305 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da1b3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.568553:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ff999839/0x268e1a602cc2305 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da1b3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.568575:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.568578:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.568581:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.569117:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af8f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280547200:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.569135:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->172031] (req 0->172031) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.569147:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->172031] (req 0->172031) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.569158:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->172031] (req 0->172031) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.569353:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:0.0:1787827204.569549:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.569563:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.569587:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.569603:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.569624:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.569635:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.569657:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) sanity-hsm.sh: write [0, 170595) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.569675:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.569717:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:0.0:1787827204.569730:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000a5e02489[3 0000000073777751 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.569735:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000000095433b 17ffffc0001001 3:0 ffff8ab3d074c2b8 0 lru 00000080:00200000:0.0:1787827204.569743:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000054f34da9 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.569746:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000a5e02489 00000080:00200000:0.0:1787827204.569748:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.569755:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.569791:0:48442:0:(rw26.c:875:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:0.0:1787827204.569797:0:48442:0:(rw26.c:933:ll_write_end()) page@000000006049e8a8[3 0000000073777751 2 1 000000002663f4b0 0 0 1] 00000080:00200000:0.0:1787827204.569800:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000060688aa2 17ffffc0001001 3:0 ffff8ab3c70cf740 1 lru 00000080:00200000:0.0:1787827204.569807:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000079259d9 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.569810:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000006049e8a8 00000080:00200000:0.0:1787827204.569811:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 2. 00000080:00200000:0.0:1787827204.569814:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.569829:0:48442:0:(rw26.c:875:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:0.0:1787827204.569834:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000010bd9166[3 0000000073777751 2 1 000000002663f4b0 0 0 2] 00000080:00200000:0.0:1787827204.569838:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000000a5de593 17ffffc0001001 3:0 ffff8ab3c70cf570 2 lru 00000080:00200000:0.0:1787827204.569844:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000095287969 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.569847:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000010bd9166 00000080:00200000:0.0:1787827204.569849:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 3. 00000080:00200000:0.0:1787827204.569851:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.569867:0:48442:0:(rw26.c:875:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:0.0:1787827204.569872:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000393b7a57[3 0000000073777751 2 1 000000002663f4b0 0 0 3] 00000080:00200000:0.0:1787827204.569876:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000abc310ea 17ffffc0001001 3:0 ffff8ab3c70cf1d0 3 lru 00000080:00200000:0.0:1787827204.569921:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000f5d93f0f 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.569924:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000393b7a57 00000080:00200000:0.0:1787827204.569926:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 4. 00000080:00200000:0.0:1787827204.569928:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.569945:0:48442:0:(rw26.c:875:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:0.0:1787827204.569951:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000006c3ae60[3 0000000073777751 2 1 000000002663f4b0 0 0 4] 00000080:00200000:0.0:1787827204.569955:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000f448806f 17ffffc0001001 3:0 ffff8ab3c70cf658 4 lru 00000080:00200000:0.0:1787827204.569963:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000b3563e1e 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.569966:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000006c3ae60 00000080:00200000:0.0:1787827204.569968:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 5. 00000080:00200000:0.0:1787827204.569970:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.569983:0:48442:0:(rw26.c:875:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:0.0:1787827204.569988:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000a245e715[3 0000000073777751 2 1 000000002663f4b0 0 0 5] 00000080:00200000:0.0:1787827204.569991:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000dc907e17 17ffffc0001001 3:0 ffff8ab3c70cf000 5 lru 00000080:00200000:0.0:1787827204.569998:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000008f104d85 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570001:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000a245e715 00000080:00200000:0.0:1787827204.570002:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 6. 00000080:00200000:0.0:1787827204.570004:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570026:0:48442:0:(rw26.c:875:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570030:0:48442:0:(rw26.c:933:ll_write_end()) page@000000000d53d285[3 0000000073777751 2 1 000000002663f4b0 0 0 6] 00000080:00200000:0.0:1787827204.570034:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000003f60899 17ffffc0001021 2:0 ffff8ab3c70cfe80 6 lru 00000080:00200000:0.0:1787827204.570041:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000027dbd3f5 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570043:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000000d53d285 00000080:00200000:0.0:1787827204.570045:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 7. 00000080:00200000:0.0:1787827204.570046:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570060:0:48442:0:(rw26.c:875:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570065:0:48442:0:(rw26.c:933:ll_write_end()) page@000000001d3686ee[3 0000000073777751 2 1 000000002663f4b0 0 0 7] 00000080:00200000:0.0:1787827204.570069:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000055b0f031 17ffffc0001001 3:0 ffff8ab3c70cfae0 7 lru 00000080:00200000:0.0:1787827204.570077:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000047a2cce5 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570080:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000001d3686ee 00000080:00200000:0.0:1787827204.570082:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 8. 00000080:00200000:0.0:1787827204.570084:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570114:0:48442:0:(rw26.c:875:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570121:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000258f43c9[3 0000000073777751 2 1 000000002663f4b0 0 0 8] 00000080:00200000:0.0:1787827204.570125:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000032512bc9 17ffffc0001001 3:0 ffff8ab3c70cfbc8 8 lru 00000080:00200000:0.0:1787827204.570131:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000005d525bd6 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570135:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000258f43c9 00000080:00200000:0.0:1787827204.570137:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 9. 00000080:00200000:0.0:1787827204.570140:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570157:0:48442:0:(rw26.c:875:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570163:0:48442:0:(rw26.c:933:ll_write_end()) page@000000007b48d20d[3 0000000073777751 2 1 000000002663f4b0 0 0 9] 00000080:00200000:0.0:1787827204.570167:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000fc509759 17ffffc0001001 3:0 ffff8ab3c70cf910 9 lru 00000080:00200000:0.0:1787827204.570174:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000293d2e1b 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570177:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000007b48d20d 00000080:00200000:0.0:1787827204.570179:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 10. 00000080:00200000:0.0:1787827204.570181:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570201:0:48442:0:(rw26.c:875:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570211:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000f1295515[3 0000000073777751 2 1 000000002663f4b0 0 0 10] 00000080:00200000:0.0:1787827204.570215:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000a3163b93 17ffffc0001001 3:0 ffff8ab3c70cf828 10 lru 00000080:00200000:0.0:1787827204.570221:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000fe4d4e2b 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570224:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000f1295515 00000080:00200000:0.0:1787827204.570225:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 11. 00000080:00200000:0.0:1787827204.570227:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570243:0:48442:0:(rw26.c:875:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570248:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000c93c2b93[3 0000000073777751 2 1 000000002663f4b0 0 0 11] 00000080:00200000:0.0:1787827204.570251:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000d5c5e94d 17ffffc0001001 3:0 ffff8ab3c70cfcb0 11 lru 00000080:00200000:0.0:1787827204.570257:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000003b490187 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570259:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000c93c2b93 00000080:00200000:0.0:1787827204.570261:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 12. 00000080:00200000:0.0:1787827204.570264:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570280:0:48442:0:(rw26.c:875:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570285:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000048f97e15[3 0000000073777751 2 1 000000002663f4b0 0 0 12] 00000080:00200000:0.0:1787827204.570288:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000001a8854fe 17ffffc0001001 3:0 ffff8ab3c70cfd98 12 lru 00000080:00200000:0.0:1787827204.570293:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ca5dd40f 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570296:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000048f97e15 00000080:00200000:0.0:1787827204.570297:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 13. 00000080:00200000:0.0:1787827204.570299:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570315:0:48442:0:(rw26.c:875:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570320:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000f976f41c[3 0000000073777751 2 1 000000002663f4b0 0 0 13] 00000080:00200000:0.0:1787827204.570323:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000020f5637 17ffffc0001001 3:0 ffff8ab3c70cf2b8 13 lru 00000080:00200000:0.0:1787827204.570328:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e73da25d 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570331:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000f976f41c 00000080:00200000:0.0:1787827204.570332:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 14. 00000080:00200000:0.0:1787827204.570335:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570350:0:48442:0:(rw26.c:875:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570356:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000096d827a6[3 0000000073777751 2 1 000000002663f4b0 0 0 14] 00000080:00200000:0.0:1787827204.570359:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000d0bfdf23 17ffffc0001001 3:0 ffff8ab3c70cf488 14 lru 00000080:00200000:0.0:1787827204.570365:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000001a063b68 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570368:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000096d827a6 00000080:00200000:0.0:1787827204.570369:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 15. 00000080:00200000:0.0:1787827204.570371:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570387:0:48442:0:(rw26.c:875:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570393:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000d15371cd[3 0000000073777751 2 1 000000002663f4b0 0 0 15] 00000080:00200000:0.0:1787827204.570396:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000060526cf7 17ffffc0001001 3:0 ffff8ab3c70cf3a0 15 lru 00000080:00200000:0.0:1787827204.570402:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000080b58699 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570404:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000d15371cd 00000080:00200000:0.0:1787827204.570405:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 16. 00000080:00200000:0.0:1787827204.570408:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570424:0:48442:0:(rw26.c:875:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570429:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000622d4164[3 0000000073777751 2 1 000000002663f4b0 0 0 16] 00000080:00200000:0.0:1787827204.570432:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000002970f41d 17ffffc0001001 3:0 ffff8ab3c70cf9f8 16 lru 00000080:00200000:0.0:1787827204.570440:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000cb35ac17 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570442:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000622d4164 00000080:00200000:0.0:1787827204.570443:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 17. 00000080:00200000:0.0:1787827204.570445:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570463:0:48442:0:(rw26.c:875:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570468:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000a14ace74[3 0000000073777751 2 1 000000002663f4b0 0 0 17] 00000080:00200000:0.0:1787827204.570471:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000f00053bb 17ffffc0001001 3:0 ffff8ab3c70cf0e8 17 lru 00000080:00200000:0.0:1787827204.570478:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000009ee7e76d 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570481:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000a14ace74 00000080:00200000:0.0:1787827204.570483:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 18. 00000080:00200000:0.0:1787827204.570485:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570508:0:48442:0:(rw26.c:875:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570513:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000016f9727d[3 0000000073777751 2 1 000000002663f4b0 0 0 18] 00000080:00200000:0.0:1787827204.570517:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000002d9da8fb 17ffffc0001001 3:0 ffff8ab3f0419e80 18 lru 00000080:00200000:0.0:1787827204.570524:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000619e871b 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570526:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000016f9727d 00000080:00200000:0.0:1787827204.570528:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 19. 00000080:00200000:0.0:1787827204.570530:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570547:0:48442:0:(rw26.c:875:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570552:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000d2440ac2[3 0000000073777751 2 1 000000002663f4b0 0 0 19] 00000080:00200000:0.0:1787827204.570555:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000aefce371 17ffffc0001001 3:0 ffff8ab3f0419ae0 19 lru 00000080:00200000:0.0:1787827204.570562:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000070ef0e9b 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570564:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000d2440ac2 00000080:00200000:0.0:1787827204.570566:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 20. 00000080:00200000:0.0:1787827204.570568:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570585:0:48442:0:(rw26.c:875:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570590:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000c10d8c71[3 0000000073777751 2 1 000000002663f4b0 0 0 20] 00000080:00200000:0.0:1787827204.570594:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000a94619eb 17ffffc0001001 3:0 ffff8ab3f0419bc8 20 lru 00000080:00200000:0.0:1787827204.570600:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ffaffc66 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570603:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000c10d8c71 00000080:00200000:0.0:1787827204.570604:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 21. 00000080:00200000:0.0:1787827204.570607:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570627:0:48442:0:(rw26.c:875:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570632:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000082c70642[3 0000000073777751 2 1 000000002663f4b0 0 0 21] 00000080:00200000:0.0:1787827204.570635:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000008a327301 17ffffc0001021 2:0 ffff8ab3f0419910 21 lru 00000080:00200000:0.0:1787827204.570643:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000003a003ce3 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570645:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000082c70642 00000080:00200000:0.0:1787827204.570647:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 22. 00000080:00200000:0.0:1787827204.570649:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570664:0:48442:0:(rw26.c:875:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570669:0:48442:0:(rw26.c:933:ll_write_end()) page@000000001338810b[3 0000000073777751 2 1 000000002663f4b0 0 0 22] 00000080:00200000:0.0:1787827204.570673:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000003c74cdf 17ffffc0001001 3:0 ffff8ab3f0419828 22 lru 00000080:00200000:0.0:1787827204.570680:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000000501893a 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570682:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000001338810b 00000080:00200000:0.0:1787827204.570684:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 23. 00000080:00200000:0.0:1787827204.570686:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570701:0:48442:0:(rw26.c:875:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570706:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000969795df[3 0000000073777751 2 1 000000002663f4b0 0 0 23] 00000080:00200000:0.0:1787827204.570710:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000005ba0809d 17ffffc0001001 3:0 ffff8ab3f0419cb0 23 lru 00000080:00200000:0.0:1787827204.570716:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ef7435e6 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570719:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000969795df 00000080:00200000:0.0:1787827204.570720:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 24. 00000080:00200000:0.0:1787827204.570722:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570741:0:48442:0:(rw26.c:875:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570746:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000fd5ccc5a[3 0000000073777751 2 1 000000002663f4b0 0 0 24] 00000080:00200000:0.0:1787827204.570750:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000243aae0f 17ffffc0001001 3:0 ffff8ab3f0419d98 24 lru 00000080:00200000:0.0:1787827204.570757:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000bbd5e12a 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570759:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000fd5ccc5a 00000080:00200000:0.0:1787827204.570761:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 25. 00000080:00200000:0.0:1787827204.570763:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570779:0:48442:0:(rw26.c:875:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570784:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000066200f4c[3 0000000073777751 2 1 000000002663f4b0 0 0 25] 00000080:00200000:0.0:1787827204.570787:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000009f2cbbbe 17ffffc0001001 3:0 ffff8ab3f04192b8 25 lru 00000080:00200000:0.0:1787827204.570795:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e80855b6 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570798:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000066200f4c 00000080:00200000:0.0:1787827204.570799:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 26. 00000080:00200000:0.0:1787827204.570801:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570817:0:48442:0:(rw26.c:875:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570822:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000050c6ea4b[3 0000000073777751 2 1 000000002663f4b0 0 0 26] 00000080:00200000:0.0:1787827204.570825:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000002a0ccefb 17ffffc0001001 3:0 ffff8ab3f0419488 26 lru 00000080:00200000:0.0:1787827204.570831:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c008ecb1 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570834:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000050c6ea4b 00000080:00200000:0.0:1787827204.570835:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 27. 00000080:00200000:0.0:1787827204.570837:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570854:0:48442:0:(rw26.c:875:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570859:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000014196b90[3 0000000073777751 2 1 000000002663f4b0 0 0 27] 00000080:00200000:0.0:1787827204.570863:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000004fa9e3c9 17ffffc0001001 3:0 ffff8ab3f04193a0 27 lru 00000080:00200000:0.0:1787827204.570869:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000fab17bac 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570872:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000014196b90 00000080:00200000:0.0:1787827204.570873:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 28. 00000080:00200000:0.0:1787827204.570876:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570919:0:48442:0:(rw26.c:875:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570924:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000a9bebfe1[3 0000000073777751 2 1 000000002663f4b0 0 0 28] 00000080:00200000:0.0:1787827204.570928:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000028c20393 17ffffc0001001 3:0 ffff8ab3f04199f8 28 lru 00000080:00200000:0.0:1787827204.570934:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000005f9a67e8 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570937:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000a9bebfe1 00000080:00200000:0.0:1787827204.570938:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 29. 00000080:00200000:0.0:1787827204.570940:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570955:0:48442:0:(rw26.c:875:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570960:0:48442:0:(rw26.c:933:ll_write_end()) page@000000003d74f02a[3 0000000073777751 2 1 000000002663f4b0 0 0 29] 00000080:00200000:0.0:1787827204.570964:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000226eb0e4 17ffffc0001001 3:0 ffff8ab3f04190e8 29 lru 00000080:00200000:0.0:1787827204.570970:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000570fb643 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.570973:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000003d74f02a 00000080:00200000:0.0:1787827204.570975:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 30. 00000080:00200000:0.0:1787827204.570977:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.570993:0:48442:0:(rw26.c:875:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:0.0:1787827204.570998:0:48442:0:(rw26.c:933:ll_write_end()) page@000000000d1e05fd[3 0000000073777751 2 1 000000002663f4b0 0 0 30] 00000080:00200000:0.0:1787827204.571001:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000007c1d0d64 17ffffc0001001 3:0 ffff8ab3f0419740 30 lru 00000080:00200000:0.0:1787827204.571007:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000b1245659 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.571010:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000000d1e05fd 00000080:00200000:0.0:1787827204.571012:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 31. 00000080:00200000:0.0:1787827204.571013:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.571032:0:48442:0:(rw26.c:875:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:0.0:1787827204.571036:0:48442:0:(rw26.c:933:ll_write_end()) page@000000003607f6a8[3 0000000073777751 2 1 000000002663f4b0 0 0 31] 00000080:00200000:0.0:1787827204.571040:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000027ef2980 17ffffc0001001 3:0 ffff8ab3f0419570 31 lru 00000080:00200000:0.0:1787827204.571046:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000075675070 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.571048:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000003607f6a8 00000080:00200000:0.0:1787827204.571050:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 32. 00000080:00200000:0.0:1787827204.571052:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.571068:0:48442:0:(rw26.c:875:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:0.0:1787827204.571073:0:48442:0:(rw26.c:933:ll_write_end()) page@000000001534200f[3 0000000073777751 2 1 000000002663f4b0 0 0 32] 00000080:00200000:0.0:1787827204.571077:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000001cbec88f 17ffffc0001001 3:0 ffff8ab3f04191d0 32 lru 00000080:00200000:0.0:1787827204.571083:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ed116e30 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.571086:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000001534200f 00000080:00200000:0.0:1787827204.571087:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 33. 00000080:00200000:0.0:1787827204.571089:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.571123:0:48442:0:(rw26.c:875:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:0.0:1787827204.571128:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000039282985[3 0000000073777751 2 1 000000002663f4b0 0 0 33] 00000080:00200000:0.0:1787827204.571132:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000a3a20deb 17ffffc0001001 3:0 ffff8ab3f0419658 33 lru 00000080:00200000:0.0:1787827204.571138:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000a0bdce3d 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.571140:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000039282985 00000080:00200000:0.0:1787827204.571142:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 34. 00000080:00200000:0.0:1787827204.571144:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 34 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.571162:0:48442:0:(rw26.c:875:ll_write_end()) pos 139264, len 4096, copied 4096 00000080:00200000:0.0:1787827204.571167:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000e1c6a106[3 0000000073777751 2 1 000000002663f4b0 0 0 34] 00000080:00200000:0.0:1787827204.571171:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000f3ec1647 17ffffc0001001 3:0 ffff8ab3f0419000 34 lru 00000080:00200000:0.0:1787827204.571177:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000002f000dcd 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.571179:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000e1c6a106 00000080:00200000:0.0:1787827204.571181:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 35. 00000080:00200000:0.0:1787827204.571183:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 35 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.571207:0:48442:0:(rw26.c:875:ll_write_end()) pos 143360, len 4096, copied 4096 00000080:00200000:0.0:1787827204.571213:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000caa006ba[3 0000000073777751 2 1 000000002663f4b0 0 0 35] 00000080:00200000:0.0:1787827204.571216:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000f4e94fb4 17ffffc0001001 3:0 ffff8ab3f0759570 35 lru 00000080:00200000:0.0:1787827204.571222:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000002161259b 35: 1< 0 - - > 2< 143360 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.571225:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000caa006ba 00000080:00200000:0.0:1787827204.571226:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 36. 00000080:00200000:0.0:1787827204.571228:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 36 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.571249:0:48442:0:(rw26.c:875:ll_write_end()) pos 147456, len 4096, copied 4096 00000080:00200000:0.0:1787827204.571254:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000f46f5587[3 0000000073777751 2 1 000000002663f4b0 0 0 36] 00000080:00200000:0.0:1787827204.571258:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000578b7d0b 17ffffc0001021 2:0 ffff8ab3f07591d0 36 lru 00000080:00200000:0.0:1787827204.571264:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000096fe10b4 36: 1< 0 - - > 2< 147456 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.571266:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000f46f5587 00000080:00200000:0.0:1787827204.571267:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 37. 00000080:00200000:0.0:1787827204.571269:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 37 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.571285:0:48442:0:(rw26.c:875:ll_write_end()) pos 151552, len 4096, copied 4096 00000080:00200000:0.0:1787827204.571290:0:48442:0:(rw26.c:933:ll_write_end()) page@000000002fa516eb[3 0000000073777751 2 1 000000002663f4b0 0 0 37] 00000080:00200000:0.0:1787827204.571293:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000454e1085 17ffffc0001001 3:0 ffff8ab3f0759658 37 lru 00000080:00200000:0.0:1787827204.571299:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ca2cdfc1 37: 1< 0 - - > 2< 151552 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.571301:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000002fa516eb 00000080:00200000:0.0:1787827204.571303:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 38. 00000080:00200000:0.0:1787827204.571305:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 38 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.571324:0:48442:0:(rw26.c:875:ll_write_end()) pos 155648, len 4096, copied 4096 00000080:00200000:0.0:1787827204.571328:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000091b50559[3 0000000073777751 2 1 000000002663f4b0 0 0 38] 00000080:00200000:0.0:1787827204.571331:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000036bd1e9a 17ffffc0001001 3:0 ffff8ab3f0759000 38 lru 00000080:00200000:0.0:1787827204.571337:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000f4118041 38: 1< 0 - - > 2< 155648 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.571339:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000091b50559 00000080:00200000:0.0:1787827204.571341:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 39. 00000080:00200000:0.0:1787827204.571343:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 39 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.571359:0:48442:0:(rw26.c:875:ll_write_end()) pos 159744, len 4096, copied 4096 00000080:00200000:0.0:1787827204.571364:0:48442:0:(rw26.c:933:ll_write_end()) page@000000003dd2f359[3 0000000073777751 2 1 000000002663f4b0 0 0 39] 00000080:00200000:0.0:1787827204.571368:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000281f2f45 17ffffc0001001 3:0 ffff8ab3f0759e80 39 lru 00000080:00200000:0.0:1787827204.571374:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000006bdd1cf1 39: 1< 0 - - > 2< 159744 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.571376:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000003dd2f359 00000080:00200000:0.0:1787827204.571378:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 40. 00000080:00200000:0.0:1787827204.571379:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 40 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.571392:0:48442:0:(rw26.c:875:ll_write_end()) pos 163840, len 4096, copied 4096 00000080:00200000:0.0:1787827204.571396:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000049dbc567[3 0000000073777751 2 1 000000002663f4b0 0 0 40] 00000080:00200000:0.0:1787827204.571400:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000000df1b76b 17ffffc0001001 3:0 ffff8ab3f0759ae0 40 lru 00000080:00200000:0.0:1787827204.571406:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000bec5328b 40: 1< 0 - - > 2< 163840 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.571409:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000049dbc567 00000080:00200000:0.0:1787827204.571410:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 41. 00000080:00200000:0.0:1787827204.571412:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 41 of 0 to 2659 bytes 00000080:00200000:0.0:1787827204.571429:0:48442:0:(rw26.c:875:ll_write_end()) pos 167936, len 2659, copied 2659 00000080:00200000:0.0:1787827204.571434:0:48442:0:(rw26.c:933:ll_write_end()) page@000000003cad7672[3 0000000073777751 2 1 000000002663f4b0 0 0 41] 00000080:00200000:0.0:1787827204.571437:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000008de3c7f2 17ffffc0001001 3:0 ffff8ab3f0759bc8 41 lru 00000080:00200000:0.0:1787827204.571443:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000042c4b2d7 41: 1< 0 - - > 2< 167936 0 0 0x0 0x0 | 00000000f5856a27 00000000cfb95bb1 > 3< 0 0 > 4< 0 0 8 2674688 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.571446:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000003cad7672 00000080:00200000:0.0:1787827204.571447:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 42. 00000080:00200000:0.0:1787827204.571451:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 42, from 0, to 2659 prio 0 00000008:00010000:0.0:1787827204.571472:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571493:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571504:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571522:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571532:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571542:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571552:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571560:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571568:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571577:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571585:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571594:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571604:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571612:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571620:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.571636:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f1239b38, count 15, dirtied 15 00000008:00010000:0.0:1787827204.571643:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571652:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571663:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571672:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571682:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571697:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571707:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571717:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571726:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571735:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571745:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571755:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571764:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571770:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571777:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.571785:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f1239b38, count 15, dirtied 15 00000008:00010000:0.0:1787827204.571789:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571794:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571800:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571805:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571810:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571815:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571820:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571829:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571834:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571839:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571844:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.571849:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.571856:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f1239b38, count 12, dirtied 12 00000080:00200000:0.0:1787827204.571858:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 42 pages 170595 bytes, tot: 170595 00000080:00200000:0.0:1787827204.571861:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) sanity-hsm.sh: write bytes 0, result: 170595 00000020:00010000:0.0:1787827204.571873:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.571876:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.571878:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f530 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.571882:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.571904:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.571916:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.571919:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f530 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.571924:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 170595 bytes 00000080:00200000:0.0:1787827204.571930:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 41] 00000080:00200000:0.0:1787827204.571936:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3b:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.571939:0:48442:0:(file.c:2240:ll_file_io_generic()) sanity-hsm.sh: 2 io complete with rc: 0, result: 170595, restart: 0 00000080:00200000:0.0:1787827204.571946:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x3b:0x0] updating i_size 170595 i_blocks 0 00000080:00200000:0.0:1787827204.571950:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 170595 00000080:00200008:0.0:1787827204.571977:0:48442:0:(file.c:479:ll_file_release()) START file sanity-hsm.sh:[0x240001b71:0x3b:0x0](ffff8ab3f1239988), flags 100001 00000002:00100000:0.0:1787827204.572012:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3d0af92c0 x1874671280546944/t17179869490(17179869490) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.572022:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.572027:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af8f00 x1874671280547584/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.572039:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af8f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280547584:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.572075:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c600 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.573131:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af8f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280547584:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.573151:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3d0af92c0 x1874671280546944/t17179869490(17179869490) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.573210:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.573227:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.573244:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.573264:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.573274:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.573281:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.573285:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.573291:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.573295:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.573305:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.573316:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.573324:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.573331:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.573335:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.573339:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.573342:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-hsm.sh, flags=8 00000080:00200008:0.0:1787827204.573351:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-hsm.sh:[0x240001b71:0x3b:0x0](ffff8ab3f1239988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.573356:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x3b:0x0](ffff8ab3f1239988),name=sanity-hsm.sh 00000002:00010000:0.0:1787827204.573368:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x3b:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.573378:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.573387:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.573396:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.573400:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x3b:0x0](ffff8ab3f1239988) 00000080:00010000:0.0:1787827204.573402:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x3b:0x0](ffff8ab3f1239988) for lock 0x268e1a602cc22fe 00000080:00010000:0.0:1787827204.573407:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc22fe from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.573410:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.573419:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.573435:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3b:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.573444:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x3b:0x0] 00000008:00010000:0.0:1787827204.573457:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.573461:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f530, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.573473:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.573485:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.573497:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.573501:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.573510:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.573527:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.573533:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x3b:0x0] updating i_size 170595 i_blocks 0 00000020:00010000:0.0:1787827204.573538:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.573541:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.573544:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f530 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.573548:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.573551:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f530 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.573558:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3b:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.573603:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.573611:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.573618:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.573627:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.573635:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.573639:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.573643:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.573647:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.573651:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.573660:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.573685:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.573689:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.573693:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.573696:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.573699:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.573711:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-krb5.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.573716:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-krb5.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.573727:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-krb5.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.573760:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2313 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.573769:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2313 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.573779:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2313 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.573793:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af9e00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280547712:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.573834:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c600 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.574957:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af9e00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280547712:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.574969:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2313 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.574980:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2313 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.574988:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2313 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.574995:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2313 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.575004:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2313 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.575011:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cea00 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.575016:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2313 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.575025:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2313 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.575034:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2313 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.575043:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.575050:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2313 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.575058:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2313 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.575075:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0af9e00 x1874671280547712/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.575089:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.575120:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.575164:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.575174:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.575182:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.575193:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.575200:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.575205:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.575208:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.575214:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.575217:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.575225:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.575235:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.575240:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.575244:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.575247:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.575250:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.575259:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-krb5.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.575262:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-krb5.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.575271:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-krb5.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.575290:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc231a lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.575298:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc231a lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.575311:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc231a lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.575324:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af9e00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280547840:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.575362:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c600 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.576322:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af9e00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280547840:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.576333:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc231a lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.576343:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc231a lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.576350:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc231a lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.576358:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc231a lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.576366:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc231a lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.576372:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cf600 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.576376:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc231a lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.576383:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc231a lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.576390:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc231a lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.576397:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.576402:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc231a lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.576409:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc231a lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.576422:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0af9e00 x1874671280547840/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.576434:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.576446:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.576502:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.576510:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.576518:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.576529:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.576537:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.576541:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.576544:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.576548:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.576552:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.576562:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.576574:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.576579:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.576583:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.576586:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.576590:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.576600:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.576605:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=sanity-krb5.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be3b00, open_flags 80c1, mode 81ed opened 0 00000080:00200000:0.0:1787827204.576614:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-krb5.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.576630:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-krb5.sh,[0x240001b71:0x3c:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.576662:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.576671:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.576690:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.576698:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af9e00 x1874671280548096/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.576708:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.576722:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af9e00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280548096:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.576828:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c600 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.578266:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af9e00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280548096:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.578281:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.578306:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.578315:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.578327:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.578338:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.578349:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.578356:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.578364:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.578374:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3d0af9e00 x1874671280548096/t17179869492(17179869492) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.578386:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0af9e00 x1874671280548096/t17179869492(17179869492) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.578399:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.578411:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.578420:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/60 (0 0) 00000080:00200000:0.0:1787827204.578424:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.578433:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=sanity-krb5.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.578460:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x3c:0x0](ffff8ab3f1239148) 00000080:00200000:0.0:1787827204.578544:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x3c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.578549:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x3c:0x0](ffff8ab3f1239148) 00000080:00010000:0.0:1787827204.578552:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x3c:0x0](ffff8ab3f1239148) for lock 0x268e1a602cc2321 00000080:00010000:0.0:1787827204.578559:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x3c:0x0](ffff8ab3f1239148) 00000080:00200008:0.0:1787827204.578567:0:48442:0:(file.c:1090:ll_file_open()) START file sanity-krb5.sh:[0x240001b71:0x3c:0x0](ffff8ab3f1239148), flags 100301 00000002:00100000:0.0:1787827204.578582:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0af9e00 x1874671280548096/t17179869492(17179869492) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.578610:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2321 from it ffff8ab3d0a11300 00010000:00010000:0.0:1787827204.578614:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.578621:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.578634:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x3c:0x0](ffff8ab3f1239148), inode mode 81ed mask 40 00000080:00200008:0.0:1787827204.578647:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-krb5.sh:[0x240001b71:0x3c:0x0](ffff8ab3f1239148), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.578650:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x3c:0x0](ffff8ab3f1239148),name=sanity-krb5.sh 00000002:00010000:0.0:1787827204.578659:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x3c:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.578666:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.578683:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.578688:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.578692:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x3c:0x0](ffff8ab3f1239148) 00000080:00010000:0.0:1787827204.578694:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x3c:0x0](ffff8ab3f1239148) for lock 0x268e1a602cc2321 00000080:00010000:0.0:1787827204.578697:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2321 from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.578699:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.578704:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.578713:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3c:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.578719:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x3c:0x0] 00000008:00010000:0.0:1787827204.578730:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.578732:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f5b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.578768:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b9cac220/0x268e1a602cc2328 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.578777:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b9cac220/0x268e1a602cc2328 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.578800:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1787827204.579987:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x3c:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787827204.579997:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.580003:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.580011:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f5b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.580018:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.580022:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f5b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.580034:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3c:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.580092:0:48442:0:(file.c:2652:do_file_write_iter()) START file sanity-krb5.sh:[0x240001b71:0x3c:0x0], ppos: 0, count: 31711 00000080:00200000:0.0:1787827204.580116:0:48442:0:(file.c:2090:ll_file_io_generic()) sanity-krb5.sh: write ppos: 0, bytes: 31711 00000080:00200000:0.0:1787827204.580123:0:48442:0:(file.c:1848:ll_io_set_mirror()) sanity-krb5.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.580130:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@0000000089e4ccfa[0x110, 1, [0x240001b71:0x3c:0x0] hash] 00000020:00200000:0.0:1787827204.580132:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 31711) 0 0 00000080:00200000:0.0:1787827204.580136:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3c:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.580145:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 7] 00020000:00200000:0.0:1787827204.580153:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 31711) 31711, 31711 00020000:00200000:0.0:1787827204.580157:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.580172:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 7] 00000008:00010000:0.0:1787827204.580181:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.580185:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f5b0, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.580193:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d2d3cc00 type 11 mode 2 res 11811161088/159 (0 32767) 00010000:00010000:0.0:1787827204.580215:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.580225:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->32767] (req 0->32767) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.580235:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->32767] (req 0->32767) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.580251:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af83c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280548352:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.580293:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.581031:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.581239:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.581248:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b9cac220/0x268e1a602cc2328 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da20e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.581257:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cf000 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.581262:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b9cac220/0x268e1a602cc2328 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da20e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.581276:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.581283:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.581315:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b9cac220/0x268e1a602cc2328 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da20e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.581369:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.581395:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b9cac220/0x268e1a602cc2328 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da20e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.581402:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b9cac220/0x268e1a602cc2328 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da20e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.581416:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.581419:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.581422:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.581848:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af83c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280548352:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.581864:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->32767] (req 0->32767) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.581877:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->32767] (req 0->32767) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.581937:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->32767] (req 0->32767) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.582261:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:0.0:1787827204.582473:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.582489:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.582511:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.582530:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.582542:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.582553:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.582574:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) sanity-krb5.sh: write [0, 31711) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.582588:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.582633:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:0.0:1787827204.582646:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000038887a13[3 000000005d097d30 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.582651:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000c4b16b71 17ffffc0001001 3:0 ffff8ab3f0759910 0 lru 00000080:00200000:0.0:1787827204.582659:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000eb3bc912 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000006a588f83 00000000b31dbd6b > 3< 0 0 > 4< 0 0 8 3325952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.582662:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000038887a13 00000080:00200000:0.0:1787827204.582664:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.582667:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.582695:0:48442:0:(rw26.c:875:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:0.0:1787827204.582701:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000ef290392[3 000000005d097d30 2 1 000000002663f4b0 0 0 1] 00000080:00200000:0.0:1787827204.582705:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000003ba54933 17ffffc0001001 3:0 ffff8ab3f0759828 1 lru 00000080:00200000:0.0:1787827204.582711:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e3aba616 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000006a588f83 00000000b31dbd6b > 3< 0 0 > 4< 0 0 8 3325952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.582714:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000ef290392 00000080:00200000:0.0:1787827204.582715:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 2. 00000080:00200000:0.0:1787827204.582718:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.582741:0:48442:0:(rw26.c:875:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:0.0:1787827204.582746:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000df2cd08f[3 000000005d097d30 2 1 000000002663f4b0 0 0 2] 00000080:00200000:0.0:1787827204.582750:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000006f3091d7 17ffffc0001001 3:0 ffff8ab3f0759cb0 2 lru 00000080:00200000:0.0:1787827204.582757:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c278fa77 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 000000006a588f83 00000000b31dbd6b > 3< 0 0 > 4< 0 0 8 3325952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.582759:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000df2cd08f 00000080:00200000:0.0:1787827204.582761:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 3. 00000080:00200000:0.0:1787827204.582763:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.582781:0:48442:0:(rw26.c:875:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:0.0:1787827204.582786:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000c6d24129[3 000000005d097d30 2 1 000000002663f4b0 0 0 3] 00000080:00200000:0.0:1787827204.582789:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000ac352837 17ffffc0001001 3:0 ffff8ab3f0759d98 3 lru 00000080:00200000:0.0:1787827204.582796:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000cc1ed2b3 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 000000006a588f83 00000000b31dbd6b > 3< 0 0 > 4< 0 0 8 3325952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.582798:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000c6d24129 00000080:00200000:0.0:1787827204.582800:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 4. 00000080:00200000:0.0:1787827204.582802:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.582817:0:48442:0:(rw26.c:875:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:0.0:1787827204.582822:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000880ef1f5[3 000000005d097d30 2 1 000000002663f4b0 0 0 4] 00000080:00200000:0.0:1787827204.582825:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000413175b4 17ffffc0001001 3:0 ffff8ab3f07592b8 4 lru 00000080:00200000:0.0:1787827204.582831:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ffbfd79a 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 000000006a588f83 00000000b31dbd6b > 3< 0 0 > 4< 0 0 8 3325952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.582834:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000880ef1f5 00000080:00200000:0.0:1787827204.582835:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 5. 00000080:00200000:0.0:1787827204.582837:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.582855:0:48442:0:(rw26.c:875:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:0.0:1787827204.582860:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000a0126ac8[3 000000005d097d30 2 1 000000002663f4b0 0 0 5] 00000080:00200000:0.0:1787827204.582864:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000085efe50c 17ffffc0001001 3:0 ffff8ab3f0759488 5 lru 00000080:00200000:0.0:1787827204.582870:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000dee1965a 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 000000006a588f83 00000000b31dbd6b > 3< 0 0 > 4< 0 0 8 3325952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.582873:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000a0126ac8 00000080:00200000:0.0:1787827204.582874:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 6. 00000080:00200000:0.0:1787827204.582877:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.582921:0:48442:0:(rw26.c:875:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:0.0:1787827204.582928:0:48442:0:(rw26.c:933:ll_write_end()) page@000000007101f572[3 000000005d097d30 2 1 000000002663f4b0 0 0 6] 00000080:00200000:0.0:1787827204.582932:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000c365d15e 17ffffc0001001 3:0 ffff8ab3f07593a0 6 lru 00000080:00200000:0.0:1787827204.582939:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000624bb0d7 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 000000006a588f83 00000000b31dbd6b > 3< 0 0 > 4< 0 0 8 3325952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.582942:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000007101f572 00000080:00200000:0.0:1787827204.582943:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 7. 00000080:00200000:0.0:1787827204.582945:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 7 of 0 to 3039 bytes 00000080:00200000:0.0:1787827204.582966:0:48442:0:(rw26.c:875:ll_write_end()) pos 28672, len 3039, copied 3039 00000080:00200000:0.0:1787827204.582972:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000806f0a0a[3 000000005d097d30 2 1 000000002663f4b0 0 0 7] 00000080:00200000:0.0:1787827204.582975:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000005e18684e 17ffffc0001001 3:0 ffff8ab3f07599f8 7 lru 00000080:00200000:0.0:1787827204.582981:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000b165acc3 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 000000006a588f83 00000000b31dbd6b > 3< 0 0 > 4< 0 0 8 3325952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.582984:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000806f0a0a 00000080:00200000:0.0:1787827204.582986:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 8. 00000080:00200000:0.0:1787827204.582990:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 8, from 0, to 3039 prio 0 00000008:00010000:0.0:1787827204.583013:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000005b104820 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.583039:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000005b104820 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.583052:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000005b104820 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.583064:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000005b104820 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.583077:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000005b104820 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.583115:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000005b104820 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.583128:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000005b104820 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.583140:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 000000005b104820 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.583157:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f12392f8, count 8, dirtied 8 00000080:00200000:0.0:1787827204.583164:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 8 pages 31711 bytes, tot: 31711 00000080:00200000:0.0:1787827204.583167:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) sanity-krb5.sh: write bytes 0, result: 31711 00000020:00010000:0.0:1787827204.583181:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.583185:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.583189:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f5b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.583195:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.583205:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.583217:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.583221:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f5b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.583228:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 31711 bytes 00000080:00200000:0.0:1787827204.583234:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 7] 00000080:00200000:0.0:1787827204.583240:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3c:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.583245:0:48442:0:(file.c:2240:ll_file_io_generic()) sanity-krb5.sh: 2 io complete with rc: 0, result: 31711, restart: 0 00000080:00200000:0.0:1787827204.583251:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x3c:0x0] updating i_size 31711 i_blocks 0 00000080:00200000:0.0:1787827204.583255:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 31711 00000080:00200008:0.0:1787827204.583277:0:48442:0:(file.c:479:ll_file_release()) START file sanity-krb5.sh:[0x240001b71:0x3c:0x0](ffff8ab3f1239148), flags 100001 00000002:00100000:0.0:1787827204.583314:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3d0af9e00 x1874671280548096/t17179869492(17179869492) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.583327:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.583333:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0af83c0 x1874671280548736/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.583348:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0af83c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280548736:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.583387:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.584229:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0af83c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280548736:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.584249:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3d0af9e00 x1874671280548096/t17179869492(17179869492) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.584308:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.584322:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.584337:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.584348:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.584357:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.584364:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.584368:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.584374:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.584379:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.584389:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.584401:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.584409:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.584416:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.584420:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.584424:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.584429:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-krb5.sh, flags=8 00000080:00200008:0.0:1787827204.584436:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-krb5.sh:[0x240001b71:0x3c:0x0](ffff8ab3f1239148), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.584441:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x3c:0x0](ffff8ab3f1239148),name=sanity-krb5.sh 00000002:00010000:0.0:1787827204.584453:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x3c:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.584462:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.584473:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.584481:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.584485:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x3c:0x0](ffff8ab3f1239148) 00000080:00010000:0.0:1787827204.584488:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x3c:0x0](ffff8ab3f1239148) for lock 0x268e1a602cc2321 00000080:00010000:0.0:1787827204.584493:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2321 from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.584497:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.584505:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.584521:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3c:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.584528:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x3c:0x0] 00000008:00010000:0.0:1787827204.584538:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.584564:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f5b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.584582:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.584595:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.584606:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.584609:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.584621:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.584632:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.584638:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x3c:0x0] updating i_size 31711 i_blocks 0 00000020:00010000:0.0:1787827204.584642:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.584645:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.584648:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f5b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.584652:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.584655:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f5b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.584663:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3c:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.584706:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.584715:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.584723:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.584733:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.584741:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.584746:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.584749:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.584754:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.584758:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.584784:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.584794:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.584799:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.584804:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.584808:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.584811:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.584822:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-lfsck.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.584827:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-lfsck.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.584839:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-lfsck.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.584871:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc2336 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.584880:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc2336 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.584919:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc2336 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.584936:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afad00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280548864:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.584981:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.585956:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afad00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280548864:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.585965:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc2336 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.585972:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc2336 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.585977:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc2336 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.585981:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc2336 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.585986:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc2336 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.585990:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0760800 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.585993:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc2336 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.585998:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc2336 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.586003:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc2336 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.586007:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.586011:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc2336 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.586015:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760800/0x268e1a602cc2336 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.586025:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0afad00 x1874671280548864/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.586034:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.586046:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.586070:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.586075:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.586080:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.586087:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.586094:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.586113:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.586116:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.586122:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.586126:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.586134:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.586145:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.586150:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.586154:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.586158:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.586160:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.586169:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-lfsck.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.586173:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-lfsck.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.586182:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-lfsck.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.586204:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc233d lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.586218:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc233d lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.586226:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc233d lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.586241:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afad00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280548992:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.586280:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.587212:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afad00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280548992:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.587223:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc233d lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.587231:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc233d lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.587238:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc233d lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.587244:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc233d lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.587252:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc233d lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.587257:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f07cf600 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.587261:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc233d lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.587267:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc233d lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.587274:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc233d lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.587280:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.587284:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc233d lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.587292:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf600/0x268e1a602cc233d lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.587304:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0afad00 x1874671280548992/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.587315:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.587325:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.587375:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.587383:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.587390:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.587400:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.587408:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.587412:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.587415:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.587420:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.587423:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.587439:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.587449:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.587454:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.587458:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.587461:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.587464:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.587475:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.587480:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=sanity-lfsck.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be3080, open_flags 80c1, mode 81ed opened 0 00000080:00200000:0.0:1787827204.587487:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-lfsck.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.587502:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-lfsck.sh,[0x240001b71:0x3d:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.587531:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.587538:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.587547:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.587554:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0afad00 x1874671280549248/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.587563:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.587576:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0afad00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280549248:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.587686:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4c6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.589027:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0afad00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280549248:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.589042:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.589058:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.589068:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.589093:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.589117:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.589128:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.589138:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.589147:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.589158:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3d0afad00 x1874671280549248/t17179869494(17179869494) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.589171:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0afad00 x1874671280549248/t17179869494(17179869494) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.589186:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.589198:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.589209:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/61 (0 0) 00000080:00200000:0.0:1787827204.589213:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.589222:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=sanity-lfsck.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.589248:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x3d:0x0](ffff8ab3f123db88) 00000080:00200000:0.0:1787827204.589321:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x3d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.589328:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x3d:0x0](ffff8ab3f123db88) 00000080:00010000:0.0:1787827204.589332:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x3d:0x0](ffff8ab3f123db88) for lock 0x268e1a602cc2344 00000080:00010000:0.0:1787827204.589340:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x3d:0x0](ffff8ab3f123db88) 00000080:00200008:0.0:1787827204.589349:0:48442:0:(file.c:1090:ll_file_open()) START file sanity-lfsck.sh:[0x240001b71:0x3d:0x0](ffff8ab3f123db88), flags 100301 00000002:00100000:0.0:1787827204.589369:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0afad00 x1874671280549248/t17179869494(17179869494) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.589407:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2344 from it ffff8ab3d0a11300 00010000:00010000:0.0:1787827204.589412:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.589422:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.589438:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x3d:0x0](ffff8ab3f123db88), inode mode 81ed mask 40 00000080:00200008:0.0:1787827204.589452:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-lfsck.sh:[0x240001b71:0x3d:0x0](ffff8ab3f123db88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.589457:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x3d:0x0](ffff8ab3f123db88),name=sanity-lfsck.sh 00000002:00010000:0.0:1787827204.589469:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x3d:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.589478:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.589489:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.589497:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.589501:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x3d:0x0](ffff8ab3f123db88) 00000080:00010000:0.0:1787827204.589504:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x3d:0x0](ffff8ab3f123db88) for lock 0x268e1a602cc2344 00000080:00010000:0.0:1787827204.589509:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2344 from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.589513:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.589521:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.589536:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.589543:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x3d:0x0] 00000008:00010000:0.0:1787827204.589556:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.589559:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f0b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.589605:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c1c91360/0x268e1a602cc234b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.589617:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c1c91360/0x268e1a602cc234b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.589657:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1787827204.590645:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x3d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787827204.590654:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.590661:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.590665:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f0b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.590670:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.590674:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f0b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.590685:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3d:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.590839:0:48442:0:(file.c:2652:do_file_write_iter()) START file sanity-lfsck.sh:[0x240001b71:0x3d:0x0], ppos: 0, count: 215293 00000080:00200000:0.0:1787827204.590848:0:48442:0:(file.c:2090:ll_file_io_generic()) sanity-lfsck.sh: write ppos: 0, bytes: 215293 00000080:00200000:0.0:1787827204.590854:0:48442:0:(file.c:1848:ll_io_set_mirror()) sanity-lfsck.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.590861:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@00000000e0745abf[0x110, 1, [0x240001b71:0x3d:0x0] hash] 00000020:00200000:0.0:1787827204.590863:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 215293) 0 0 00000080:00200000:0.0:1787827204.590867:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.590876:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 52] 00020000:00200000:0.0:1787827204.590910:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 215293) 215293, 215293 00020000:00200000:0.0:1787827204.590915:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.590929:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 52] 00000008:00010000:0.0:1787827204.590939:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.590943:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f0b0, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.590951:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3c7ca6800 type 11 mode 2 res 10737419264/159 (0 217087) 00010000:00010000:0.0:1787827204.590973:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.590984:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->217087] (req 0->217087) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.590994:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->217087] (req 0->217087) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.591010:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aae940 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280549504:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.591051:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ca80 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.591850:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.592052:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.592063:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c1c91360/0x268e1a602cc234b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da269 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.592075:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6a97c00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.592081:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c1c91360/0x268e1a602cc234b lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da269 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.592109:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.592118:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.592195:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c1c91360/0x268e1a602cc234b lrc: 2/0,0 mode: --/PR res: [0x280000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da269 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.592206:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.592230:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c1c91360/0x268e1a602cc234b lrc: 1/0,0 mode: --/PR res: [0x280000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da269 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.592240:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c1c91360/0x268e1a602cc234b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da269 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.592258:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.592262:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.592266:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.592724:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aae940 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280549504:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.592735:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->217087] (req 0->217087) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.592743:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->217087] (req 0->217087) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.592750:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->217087] (req 0->217087) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.593046:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:0.0:1787827204.593258:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.593267:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.593277:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.593290:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.593312:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.593318:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.593335:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) sanity-lfsck.sh: write [0, 215293) append 0 append lockpos 0 00000080:00200000:0.0:1787827204.593346:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593379:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593388:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000b98edd0f[3 00000000d9685c15 2 1 000000002663f4b0 0 0 0] 00000080:00200000:0.0:1787827204.593391:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000184f7c51 17ffffc0001001 3:0 ffff8ab3f07590e8 0 lru 00000080:00200000:0.0:1787827204.593397:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000071dd7826 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593399:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000b98edd0f 00000080:00200000:0.0:1787827204.593400:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1787827204.593403:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593432:0:48442:0:(rw26.c:875:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593436:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000fc39ca35[3 00000000d9685c15 2 1 000000002663f4b0 0 0 1] 00000080:00200000:0.0:1787827204.593438:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000008897606f 17ffffc0001021 2:0 ffff8ab3f0759740 1 lru 00000080:00200000:0.0:1787827204.593442:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000a56f6177 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593444:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000fc39ca35 00000080:00200000:0.0:1787827204.593445:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 2. 00000080:00200000:0.0:1787827204.593446:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593466:0:48442:0:(rw26.c:875:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593469:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000af375575[3 00000000d9685c15 2 1 000000002663f4b0 0 0 2] 00000080:00200000:0.0:1787827204.593471:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000005a37c755 17ffffc0001001 3:0 ffff8ab3c865bae0 2 lru 00000080:00200000:0.0:1787827204.593475:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000002e8508c6 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593477:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000af375575 00000080:00200000:0.0:1787827204.593478:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 3. 00000080:00200000:0.0:1787827204.593479:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593488:0:48442:0:(rw26.c:875:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593491:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000ce0be512[3 00000000d9685c15 2 1 000000002663f4b0 0 0 3] 00000080:00200000:0.0:1787827204.593493:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000ebc0e7ae 17ffffc0001001 3:0 ffff8ab3c865bbc8 3 lru 00000080:00200000:0.0:1787827204.593497:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000070bb7893 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593499:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000ce0be512 00000080:00200000:0.0:1787827204.593499:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 4. 00000080:00200000:0.0:1787827204.593500:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593509:0:48442:0:(rw26.c:875:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593511:0:48442:0:(rw26.c:933:ll_write_end()) page@000000003272bbbf[3 00000000d9685c15 2 1 000000002663f4b0 0 0 4] 00000080:00200000:0.0:1787827204.593513:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000c5ae699a 17ffffc0001001 3:0 ffff8ab3c865b910 4 lru 00000080:00200000:0.0:1787827204.593517:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c574b1b0 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593518:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000003272bbbf 00000080:00200000:0.0:1787827204.593519:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 5. 00000080:00200000:0.0:1787827204.593520:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593529:0:48442:0:(rw26.c:875:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593531:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000b9b91634[3 00000000d9685c15 2 1 000000002663f4b0 0 0 5] 00000080:00200000:0.0:1787827204.593533:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000008969bf59 17ffffc0001001 3:0 ffff8ab3c865b828 5 lru 00000080:00200000:0.0:1787827204.593537:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000033b5585a 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593539:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000b9b91634 00000080:00200000:0.0:1787827204.593539:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 6. 00000080:00200000:0.0:1787827204.593540:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593549:0:48442:0:(rw26.c:875:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593551:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000096fe1eda[3 00000000d9685c15 2 1 000000002663f4b0 0 0 6] 00000080:00200000:0.0:1787827204.593553:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000c5f381cf 17ffffc0001001 3:0 ffff8ab3c865bcb0 6 lru 00000080:00200000:0.0:1787827204.593557:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000fee55040 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593558:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000096fe1eda 00000080:00200000:0.0:1787827204.593558:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 7. 00000080:00200000:0.0:1787827204.593559:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593568:0:48442:0:(rw26.c:875:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593570:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000f0b679c9[3 00000000d9685c15 2 1 000000002663f4b0 0 0 7] 00000080:00200000:0.0:1787827204.593572:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000008ecc03c2 17ffffc0001001 3:0 ffff8ab3c865bd98 7 lru 00000080:00200000:0.0:1787827204.593576:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e0b1bb85 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593577:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000f0b679c9 00000080:00200000:0.0:1787827204.593578:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 8. 00000080:00200000:0.0:1787827204.593579:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593587:0:48442:0:(rw26.c:875:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593590:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000007a0e64b[3 00000000d9685c15 2 1 000000002663f4b0 0 0 8] 00000080:00200000:0.0:1787827204.593592:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000b9bf5762 17ffffc0001001 3:0 ffff8ab3c865b2b8 8 lru 00000080:00200000:0.0:1787827204.593595:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c18686f0 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593596:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000007a0e64b 00000080:00200000:0.0:1787827204.593597:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 9. 00000080:00200000:0.0:1787827204.593598:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593608:0:48442:0:(rw26.c:875:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593610:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000962bb890[3 00000000d9685c15 2 1 000000002663f4b0 0 0 9] 00000080:00200000:0.0:1787827204.593612:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000113c86c2 17ffffc0001001 3:0 ffff8ab3c865b488 9 lru 00000080:00200000:0.0:1787827204.593616:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000537ac785 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593618:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000962bb890 00000080:00200000:0.0:1787827204.593619:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 10. 00000080:00200000:0.0:1787827204.593620:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593628:0:48442:0:(rw26.c:875:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593631:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000034686787[3 00000000d9685c15 2 1 000000002663f4b0 0 0 10] 00000080:00200000:0.0:1787827204.593633:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000c870402d 17ffffc0001001 3:0 ffff8ab3c865b3a0 10 lru 00000080:00200000:0.0:1787827204.593637:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000cb596b1d 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593638:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000034686787 00000080:00200000:0.0:1787827204.593639:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 11. 00000080:00200000:0.0:1787827204.593640:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593648:0:48442:0:(rw26.c:875:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593651:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000906332c8[3 00000000d9685c15 2 1 000000002663f4b0 0 0 11] 00000080:00200000:0.0:1787827204.593653:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000152a10ea 17ffffc0001001 3:0 ffff8ab3c865b9f8 11 lru 00000080:00200000:0.0:1787827204.593656:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000002e90e598 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593657:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000906332c8 00000080:00200000:0.0:1787827204.593658:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 12. 00000080:00200000:0.0:1787827204.593660:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593668:0:48442:0:(rw26.c:875:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593670:0:48442:0:(rw26.c:933:ll_write_end()) page@000000006be756e1[3 00000000d9685c15 2 1 000000002663f4b0 0 0 12] 00000080:00200000:0.0:1787827204.593672:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000091a428e7 17ffffc0001001 3:0 ffff8ab3c865b0e8 12 lru 00000080:00200000:0.0:1787827204.593676:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000496d9eee 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593677:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000006be756e1 00000080:00200000:0.0:1787827204.593678:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 13. 00000080:00200000:0.0:1787827204.593679:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593687:0:48442:0:(rw26.c:875:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593690:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000013aff884[3 00000000d9685c15 2 1 000000002663f4b0 0 0 13] 00000080:00200000:0.0:1787827204.593692:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000088586c23 17ffffc0001001 3:0 ffff8ab3c865b740 13 lru 00000080:00200000:0.0:1787827204.593696:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000004949543d 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593697:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000013aff884 00000080:00200000:0.0:1787827204.593698:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 14. 00000080:00200000:0.0:1787827204.593699:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593707:0:48442:0:(rw26.c:875:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593710:0:48442:0:(rw26.c:933:ll_write_end()) page@000000006b54b6b7[3 00000000d9685c15 2 1 000000002663f4b0 0 0 14] 00000080:00200000:0.0:1787827204.593711:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000ead9b979 17ffffc0001001 3:0 ffff8ab3c865b570 14 lru 00000080:00200000:0.0:1787827204.593715:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000000e992090 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593716:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000006b54b6b7 00000080:00200000:0.0:1787827204.593717:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 15. 00000080:00200000:0.0:1787827204.593717:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593726:0:48442:0:(rw26.c:875:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593728:0:48442:0:(rw26.c:933:ll_write_end()) page@000000005b6f6ca3[3 00000000d9685c15 2 1 000000002663f4b0 0 0 15] 00000080:00200000:0.0:1787827204.593730:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000a2dde738 17ffffc0001001 3:0 ffff8ab3c865b1d0 15 lru 00000080:00200000:0.0:1787827204.593733:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000074cd2077 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593734:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000005b6f6ca3 00000080:00200000:0.0:1787827204.593735:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 16. 00000080:00200000:0.0:1787827204.593736:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593747:0:48442:0:(rw26.c:875:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593750:0:48442:0:(rw26.c:933:ll_write_end()) page@000000009cb70160[3 00000000d9685c15 2 1 000000002663f4b0 0 0 16] 00000080:00200000:0.0:1787827204.593752:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000da23699a 17ffffc0001021 2:0 ffff8ab3c865b658 16 lru 00000080:00200000:0.0:1787827204.593756:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ed3b5566 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593757:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000009cb70160 00000080:00200000:0.0:1787827204.593758:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 17. 00000080:00200000:0.0:1787827204.593760:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593768:0:48442:0:(rw26.c:875:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593771:0:48442:0:(rw26.c:933:ll_write_end()) page@000000009a9cdd32[3 00000000d9685c15 2 1 000000002663f4b0 0 0 17] 00000080:00200000:0.0:1787827204.593773:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000fbc71a2c 17ffffc0001001 3:0 ffff8ab3c865b000 17 lru 00000080:00200000:0.0:1787827204.593777:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000005bbdd0cc 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593779:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000009a9cdd32 00000080:00200000:0.0:1787827204.593779:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 18. 00000080:00200000:0.0:1787827204.593780:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593789:0:48442:0:(rw26.c:875:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593792:0:48442:0:(rw26.c:933:ll_write_end()) page@000000006970c400[3 00000000d9685c15 2 1 000000002663f4b0 0 0 18] 00000080:00200000:0.0:1787827204.593794:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000043fd156c 17ffffc0001001 3:0 ffff8ab3c865be80 18 lru 00000080:00200000:0.0:1787827204.593798:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000001b835a0d 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593800:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000006970c400 00000080:00200000:0.0:1787827204.593800:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 19. 00000080:00200000:0.0:1787827204.593801:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593816:0:48442:0:(rw26.c:875:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593820:0:48442:0:(rw26.c:933:ll_write_end()) page@000000004982e9ff[3 00000000d9685c15 2 1 000000002663f4b0 0 0 19] 00000080:00200000:0.0:1787827204.593822:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000da782d63 17ffffc0001001 3:0 ffff8ab3c769de80 19 lru 00000080:00200000:0.0:1787827204.593826:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000dda3f1ae 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593828:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000004982e9ff 00000080:00200000:0.0:1787827204.593829:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 20. 00000080:00200000:0.0:1787827204.593830:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593841:0:48442:0:(rw26.c:875:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593844:0:48442:0:(rw26.c:933:ll_write_end()) page@000000003454155b[3 00000000d9685c15 2 1 000000002663f4b0 0 0 20] 00000080:00200000:0.0:1787827204.593847:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000007369def1 17ffffc0001001 3:0 ffff8ab3c769dae0 20 lru 00000080:00200000:0.0:1787827204.593851:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000002b6e58ac 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593853:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000003454155b 00000080:00200000:0.0:1787827204.593854:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 21. 00000080:00200000:0.0:1787827204.593856:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593869:0:48442:0:(rw26.c:875:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593874:0:48442:0:(rw26.c:933:ll_write_end()) page@000000004f90ccb7[3 00000000d9685c15 2 1 000000002663f4b0 0 0 21] 00000080:00200000:0.0:1787827204.593878:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000066e51db7 17ffffc0001001 3:0 ffff8ab3c769dbc8 21 lru 00000080:00200000:0.0:1787827204.593917:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000059aaa110 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593923:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000004f90ccb7 00000080:00200000:0.0:1787827204.593925:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 22. 00000080:00200000:0.0:1787827204.593928:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593947:0:48442:0:(rw26.c:875:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593952:0:48442:0:(rw26.c:933:ll_write_end()) page@000000005b61f890[3 00000000d9685c15 2 1 000000002663f4b0 0 0 22] 00000080:00200000:0.0:1787827204.593955:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000d0580d57 17ffffc0001001 3:0 ffff8ab3c769d910 22 lru 00000080:00200000:0.0:1787827204.593961:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ca0f4431 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593963:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000005b61f890 00000080:00200000:0.0:1787827204.593964:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 23. 00000080:00200000:0.0:1787827204.593965:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.593984:0:48442:0:(rw26.c:875:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:0.0:1787827204.593988:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000046b5df5d[3 00000000d9685c15 2 1 000000002663f4b0 0 0 23] 00000080:00200000:0.0:1787827204.593990:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000045ef36db 17ffffc0001001 3:0 ffff8ab3c769d828 23 lru 00000080:00200000:0.0:1787827204.593995:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e2b30fcc 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.593997:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000046b5df5d 00000080:00200000:0.0:1787827204.593998:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 24. 00000080:00200000:0.0:1787827204.594000:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594012:0:48442:0:(rw26.c:875:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594016:0:48442:0:(rw26.c:933:ll_write_end()) page@000000009a02216e[3 00000000d9685c15 2 1 000000002663f4b0 0 0 24] 00000080:00200000:0.0:1787827204.594019:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000003e3f6abc 17ffffc0001001 3:0 ffff8ab3c769dcb0 24 lru 00000080:00200000:0.0:1787827204.594024:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000029dcec1f 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594026:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000009a02216e 00000080:00200000:0.0:1787827204.594027:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 25. 00000080:00200000:0.0:1787827204.594029:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594043:0:48442:0:(rw26.c:875:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594048:0:48442:0:(rw26.c:933:ll_write_end()) page@000000009ddd762f[3 00000000d9685c15 2 1 000000002663f4b0 0 0 25] 00000080:00200000:0.0:1787827204.594051:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000008d642c53 17ffffc0001001 3:0 ffff8ab3c769dd98 25 lru 00000080:00200000:0.0:1787827204.594057:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000061705919 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594060:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000009ddd762f 00000080:00200000:0.0:1787827204.594061:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 26. 00000080:00200000:0.0:1787827204.594063:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594079:0:48442:0:(rw26.c:875:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594083:0:48442:0:(rw26.c:933:ll_write_end()) page@000000001597100f[3 00000000d9685c15 2 1 000000002663f4b0 0 0 26] 00000080:00200000:0.0:1787827204.594085:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000855a99a7 17ffffc0001001 3:0 ffff8ab3c769d2b8 26 lru 00000080:00200000:0.0:1787827204.594089:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000be4864b4 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594091:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000001597100f 00000080:00200000:0.0:1787827204.594091:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 27. 00000080:00200000:0.0:1787827204.594093:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594119:0:48442:0:(rw26.c:875:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594123:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000011519dbb[3 00000000d9685c15 2 1 000000002663f4b0 0 0 27] 00000080:00200000:0.0:1787827204.594124:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000651b0398 17ffffc0001001 3:0 ffff8ab3c769d488 27 lru 00000080:00200000:0.0:1787827204.594129:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000afa63004 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594130:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000011519dbb 00000080:00200000:0.0:1787827204.594131:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 28. 00000080:00200000:0.0:1787827204.594133:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594142:0:48442:0:(rw26.c:875:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594145:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000c1df3d27[3 00000000d9685c15 2 1 000000002663f4b0 0 0 28] 00000080:00200000:0.0:1787827204.594147:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000043abbc3b 17ffffc0001001 3:0 ffff8ab3c769d3a0 28 lru 00000080:00200000:0.0:1787827204.594151:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000005805897b 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594153:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000c1df3d27 00000080:00200000:0.0:1787827204.594153:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 29. 00000080:00200000:0.0:1787827204.594155:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594163:0:48442:0:(rw26.c:875:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594165:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000a852a7c6[3 00000000d9685c15 2 1 000000002663f4b0 0 0 29] 00000080:00200000:0.0:1787827204.594167:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000a4fa0a30 17ffffc0001001 3:0 ffff8ab3c769d9f8 29 lru 00000080:00200000:0.0:1787827204.594170:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c9a87c06 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594172:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000a852a7c6 00000080:00200000:0.0:1787827204.594173:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 30. 00000080:00200000:0.0:1787827204.594174:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594184:0:48442:0:(rw26.c:875:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594187:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000ba16e55f[3 00000000d9685c15 2 1 000000002663f4b0 0 0 30] 00000080:00200000:0.0:1787827204.594189:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000096ddc03d 17ffffc0001001 3:0 ffff8ab3c769d0e8 30 lru 00000080:00200000:0.0:1787827204.594194:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000d87ba5c5 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594197:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000ba16e55f 00000080:00200000:0.0:1787827204.594198:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 31. 00000080:00200000:0.0:1787827204.594200:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594218:0:48442:0:(rw26.c:875:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594223:0:48442:0:(rw26.c:933:ll_write_end()) page@000000003a4b9f31[3 00000000d9685c15 2 1 000000002663f4b0 0 0 31] 00000080:00200000:0.0:1787827204.594227:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000073a6a14c 17ffffc0001021 2:0 ffff8ab3c769d740 31 lru 00000080:00200000:0.0:1787827204.594233:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e1188723 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594236:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000003a4b9f31 00000080:00200000:0.0:1787827204.594237:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 32. 00000080:00200000:0.0:1787827204.594239:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594252:0:48442:0:(rw26.c:875:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594255:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000be1b8731[3 00000000d9685c15 2 1 000000002663f4b0 0 0 32] 00000080:00200000:0.0:1787827204.594257:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000046d2f12b 17ffffc0001001 3:0 ffff8ab3c769d570 32 lru 00000080:00200000:0.0:1787827204.594261:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000009eac07ad 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594263:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000be1b8731 00000080:00200000:0.0:1787827204.594264:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 33. 00000080:00200000:0.0:1787827204.594265:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594273:0:48442:0:(rw26.c:875:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594276:0:48442:0:(rw26.c:933:ll_write_end()) page@000000007a539471[3 00000000d9685c15 2 1 000000002663f4b0 0 0 33] 00000080:00200000:0.0:1787827204.594277:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000e781d6cc 17ffffc0001001 3:0 ffff8ab3c769d1d0 33 lru 00000080:00200000:0.0:1787827204.594281:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000006c0d2f55 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594282:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000007a539471 00000080:00200000:0.0:1787827204.594283:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 34. 00000080:00200000:0.0:1787827204.594285:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 34 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594293:0:48442:0:(rw26.c:875:ll_write_end()) pos 139264, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594295:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000fdfa4e15[3 00000000d9685c15 2 1 000000002663f4b0 0 0 34] 00000080:00200000:0.0:1787827204.594298:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000003faf4050 17ffffc0001001 3:0 ffff8ab3c769d658 34 lru 00000080:00200000:0.0:1787827204.594302:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000005f0bc78f 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594303:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000fdfa4e15 00000080:00200000:0.0:1787827204.594304:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 35. 00000080:00200000:0.0:1787827204.594305:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 35 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594313:0:48442:0:(rw26.c:875:ll_write_end()) pos 143360, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594316:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000421971a0[3 00000000d9685c15 2 1 000000002663f4b0 0 0 35] 00000080:00200000:0.0:1787827204.594317:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000b1a86329 17ffffc0001001 3:0 ffff8ab3c769d000 35 lru 00000080:00200000:0.0:1787827204.594321:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000009aa34d8 35: 1< 0 - - > 2< 143360 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594322:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000421971a0 00000080:00200000:0.0:1787827204.594323:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 36. 00000080:00200000:0.0:1787827204.594325:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 36 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594340:0:48442:0:(rw26.c:875:ll_write_end()) pos 147456, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594342:0:48442:0:(rw26.c:933:ll_write_end()) page@000000009dc9e3f1[3 00000000d9685c15 2 1 000000002663f4b0 0 0 36] 00000080:00200000:0.0:1787827204.594345:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000009b679039 17ffffc0001001 3:0 ffff8ab3d0904570 36 lru 00000080:00200000:0.0:1787827204.594348:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ba46d473 36: 1< 0 - - > 2< 147456 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594350:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000009dc9e3f1 00000080:00200000:0.0:1787827204.594351:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 37. 00000080:00200000:0.0:1787827204.594352:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 37 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594361:0:48442:0:(rw26.c:875:ll_write_end()) pos 151552, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594364:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000dbdc270f[3 00000000d9685c15 2 1 000000002663f4b0 0 0 37] 00000080:00200000:0.0:1787827204.594366:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000b45e9717 17ffffc0001001 3:0 ffff8ab3d09041d0 37 lru 00000080:00200000:0.0:1787827204.594369:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e12f49a0 37: 1< 0 - - > 2< 151552 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594371:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000dbdc270f 00000080:00200000:0.0:1787827204.594372:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 38. 00000080:00200000:0.0:1787827204.594373:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 38 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594381:0:48442:0:(rw26.c:875:ll_write_end()) pos 155648, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594384:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000338c5042[3 00000000d9685c15 2 1 000000002663f4b0 0 0 38] 00000080:00200000:0.0:1787827204.594386:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000087c34b58 17ffffc0001001 3:0 ffff8ab3d0904658 38 lru 00000080:00200000:0.0:1787827204.594390:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000a05a7aa7 38: 1< 0 - - > 2< 155648 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594391:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000338c5042 00000080:00200000:0.0:1787827204.594393:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 39. 00000080:00200000:0.0:1787827204.594394:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 39 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594402:0:48442:0:(rw26.c:875:ll_write_end()) pos 159744, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594405:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000afba80fb[3 00000000d9685c15 2 1 000000002663f4b0 0 0 39] 00000080:00200000:0.0:1787827204.594407:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000002f287f7 17ffffc0001001 3:0 ffff8ab3d0904000 39 lru 00000080:00200000:0.0:1787827204.594411:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000008a20fd41 39: 1< 0 - - > 2< 159744 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594412:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000afba80fb 00000080:00200000:0.0:1787827204.594413:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 40. 00000080:00200000:0.0:1787827204.594414:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 40 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594423:0:48442:0:(rw26.c:875:ll_write_end()) pos 163840, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594425:0:48442:0:(rw26.c:933:ll_write_end()) page@000000001fa8288f[3 00000000d9685c15 2 1 000000002663f4b0 0 0 40] 00000080:00200000:0.0:1787827204.594427:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000c91edc50 17ffffc0001001 3:0 ffff8ab3d0904e80 40 lru 00000080:00200000:0.0:1787827204.594430:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ecb4d09d 40: 1< 0 - - > 2< 163840 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594431:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000001fa8288f 00000080:00200000:0.0:1787827204.594432:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 41. 00000080:00200000:0.0:1787827204.594433:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 41 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594442:0:48442:0:(rw26.c:875:ll_write_end()) pos 167936, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594444:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000337e1bfc[3 00000000d9685c15 2 1 000000002663f4b0 0 0 41] 00000080:00200000:0.0:1787827204.594446:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000006a25fc27 17ffffc0001001 3:0 ffff8ab3d0904ae0 41 lru 00000080:00200000:0.0:1787827204.594449:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000003fa82adb 41: 1< 0 - - > 2< 167936 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594451:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000337e1bfc 00000080:00200000:0.0:1787827204.594451:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 42. 00000080:00200000:0.0:1787827204.594452:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 42 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594460:0:48442:0:(rw26.c:875:ll_write_end()) pos 172032, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594463:0:48442:0:(rw26.c:933:ll_write_end()) page@000000000f93f1e4[3 00000000d9685c15 2 1 000000002663f4b0 0 0 42] 00000080:00200000:0.0:1787827204.594465:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000ad31928e 17ffffc0001001 3:0 ffff8ab3d0904bc8 42 lru 00000080:00200000:0.0:1787827204.594468:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000084f9e8fb 42: 1< 0 - - > 2< 172032 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594469:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000000f93f1e4 00000080:00200000:0.0:1787827204.594470:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 43. 00000080:00200000:0.0:1787827204.594471:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 43 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594479:0:48442:0:(rw26.c:875:ll_write_end()) pos 176128, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594482:0:48442:0:(rw26.c:933:ll_write_end()) page@000000008f616856[3 00000000d9685c15 2 1 000000002663f4b0 0 0 43] 00000080:00200000:0.0:1787827204.594484:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000035532d65 17ffffc0001001 3:0 ffff8ab3d0904910 43 lru 00000080:00200000:0.0:1787827204.594488:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000cace8d73 43: 1< 0 - - > 2< 176128 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594489:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000008f616856 00000080:00200000:0.0:1787827204.594490:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 44. 00000080:00200000:0.0:1787827204.594492:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 44 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594501:0:48442:0:(rw26.c:875:ll_write_end()) pos 180224, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594504:0:48442:0:(rw26.c:933:ll_write_end()) page@000000005dc104f2[3 00000000d9685c15 2 1 000000002663f4b0 0 0 44] 00000080:00200000:0.0:1787827204.594506:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000e6ccab7f 17ffffc0001001 3:0 ffff8ab3d0904828 44 lru 00000080:00200000:0.0:1787827204.594509:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000000e6c457b 44: 1< 0 - - > 2< 180224 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594511:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000005dc104f2 00000080:00200000:0.0:1787827204.594512:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 45. 00000080:00200000:0.0:1787827204.594513:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 45 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594521:0:48442:0:(rw26.c:875:ll_write_end()) pos 184320, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594523:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000436137a7[3 00000000d9685c15 2 1 000000002663f4b0 0 0 45] 00000080:00200000:0.0:1787827204.594525:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000008054ee98 17ffffc0001001 3:0 ffff8ab3d0904cb0 45 lru 00000080:00200000:0.0:1787827204.594529:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000828d6e57 45: 1< 0 - - > 2< 184320 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594530:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000436137a7 00000080:00200000:0.0:1787827204.594531:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 46. 00000080:00200000:0.0:1787827204.594532:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 46 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594544:0:48442:0:(rw26.c:875:ll_write_end()) pos 188416, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594546:0:48442:0:(rw26.c:933:ll_write_end()) page@000000003bc3a3ea[3 00000000d9685c15 2 1 000000002663f4b0 0 0 46] 00000080:00200000:0.0:1787827204.594548:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000096097c90 17ffffc0001021 2:0 ffff8ab3d0904d98 46 lru 00000080:00200000:0.0:1787827204.594552:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000198e9967 46: 1< 0 - - > 2< 188416 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594554:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000003bc3a3ea 00000080:00200000:0.0:1787827204.594555:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 47. 00000080:00200000:0.0:1787827204.594556:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 47 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594569:0:48442:0:(rw26.c:875:ll_write_end()) pos 192512, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594573:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000059cc12df[3 00000000d9685c15 2 1 000000002663f4b0 0 0 47] 00000080:00200000:0.0:1787827204.594577:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000007ca5e0f5 17ffffc0001001 3:0 ffff8ab3d09042b8 47 lru 00000080:00200000:0.0:1787827204.594582:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c615f2b8 47: 1< 0 - - > 2< 192512 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594584:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000059cc12df 00000080:00200000:0.0:1787827204.594585:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 48. 00000080:00200000:0.0:1787827204.594586:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 48 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594598:0:48442:0:(rw26.c:875:ll_write_end()) pos 196608, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594601:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000093e16e26[3 00000000d9685c15 2 1 000000002663f4b0 0 0 48] 00000080:00200000:0.0:1787827204.594603:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000ffa64458 17ffffc0001001 3:0 ffff8ab3d0904488 48 lru 00000080:00200000:0.0:1787827204.594607:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000437a305f 48: 1< 0 - - > 2< 196608 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594608:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000093e16e26 00000080:00200000:0.0:1787827204.594609:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 49. 00000080:00200000:0.0:1787827204.594610:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 49 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594618:0:48442:0:(rw26.c:875:ll_write_end()) pos 200704, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594621:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000060776733[3 00000000d9685c15 2 1 000000002663f4b0 0 0 49] 00000080:00200000:0.0:1787827204.594623:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000004b1bb5de 17ffffc0001001 3:0 ffff8ab3d09043a0 49 lru 00000080:00200000:0.0:1787827204.594626:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000009212c798 49: 1< 0 - - > 2< 200704 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594628:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000060776733 00000080:00200000:0.0:1787827204.594629:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 50. 00000080:00200000:0.0:1787827204.594630:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 50 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594638:0:48442:0:(rw26.c:875:ll_write_end()) pos 204800, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594640:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000739ceed8[3 00000000d9685c15 2 1 000000002663f4b0 0 0 50] 00000080:00200000:0.0:1787827204.594642:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000b18e57f9 17ffffc0001001 3:0 ffff8ab3d09049f8 50 lru 00000080:00200000:0.0:1787827204.594646:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000aea24b84 50: 1< 0 - - > 2< 204800 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594647:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000739ceed8 00000080:00200000:0.0:1787827204.594647:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 51. 00000080:00200000:0.0:1787827204.594649:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 51 of 0 to 4096 bytes 00000080:00200000:0.0:1787827204.594658:0:48442:0:(rw26.c:875:ll_write_end()) pos 208896, len 4096, copied 4096 00000080:00200000:0.0:1787827204.594660:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000036a035d6[3 00000000d9685c15 2 1 000000002663f4b0 0 0 51] 00000080:00200000:0.0:1787827204.594662:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000f34aa310 17ffffc0001001 3:0 ffff8ab3d09040e8 51 lru 00000080:00200000:0.0:1787827204.594665:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c54b4500 51: 1< 0 - - > 2< 208896 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594666:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000036a035d6 00000080:00200000:0.0:1787827204.594667:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 52. 00000080:00200000:0.0:1787827204.594668:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 52 of 0 to 2301 bytes 00000080:00200000:0.0:1787827204.594682:0:48442:0:(rw26.c:875:ll_write_end()) pos 212992, len 2301, copied 2301 00000080:00200000:0.0:1787827204.594686:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000032334385[3 00000000d9685c15 2 1 000000002663f4b0 0 0 52] 00000080:00200000:0.0:1787827204.594690:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000006799ffe7 17ffffc0001001 3:0 ffff8ab3d0904740 52 lru 00000080:00200000:0.0:1787827204.594694:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ddd861d3 52: 1< 0 - - > 2< 212992 0 0 0x0 0x0 | 00000000f5856a27 00000000813414a9 > 3< 0 0 > 4< 0 0 8 2478080 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1787827204.594696:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000032334385 00000080:00200000:0.0:1787827204.594697:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 53. 00000080:00200000:0.0:1787827204.594700:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 53, from 0, to 2301 prio 0 00000008:00010000:0.0:1787827204.594729:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594745:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594751:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594756:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594761:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594766:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594771:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594776:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594786:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594791:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594796:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594801:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594806:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594811:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594815:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.594828:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f123dd38, count 15, dirtied 15 00000008:00010000:0.0:1787827204.594832:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594842:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594847:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594852:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594857:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594862:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594867:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594872:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594877:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594882:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594912:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594920:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594926:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594931:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594936:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.594944:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f123dd38, count 15, dirtied 15 00000008:00010000:0.0:1787827204.594947:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594952:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594957:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594966:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594972:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594977:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594982:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594987:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594992:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.594996:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.595001:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.595006:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.595012:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.595018:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.595023:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.595030:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f123dd38, count 15, dirtied 15 00000008:00010000:0.0:1787827204.595033:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.595038:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.595042:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.595047:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.595052:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.595061:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.595067:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.595072:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000d91e149d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827204.595077:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f123dd38, count 8, dirtied 8 00000080:00200000:0.0:1787827204.595080:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 53 pages 215293 bytes, tot: 215293 00000080:00200000:0.0:1787827204.595083:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) sanity-lfsck.sh: write bytes 0, result: 215293 00000020:00010000:0.0:1787827204.595108:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.595111:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.595113:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f0b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1787827204.595117:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.595123:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1787827204.595131:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.595133:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f0b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.595139:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 215293 bytes 00000080:00200000:0.0:1787827204.595144:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 52] 00000080:00200000:0.0:1787827204.595150:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3d:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.595153:0:48442:0:(file.c:2240:ll_file_io_generic()) sanity-lfsck.sh: 2 io complete with rc: 0, result: 215293, restart: 0 00000080:00200000:0.0:1787827204.595159:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x3d:0x0] updating i_size 215293 i_blocks 0 00000080:00200000:0.0:1787827204.595163:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 215293 00000080:00200008:0.0:1787827204.595186:0:48442:0:(file.c:479:ll_file_release()) START file sanity-lfsck.sh:[0x240001b71:0x3d:0x0](ffff8ab3f123db88), flags 100001 00000002:00100000:0.0:1787827204.595226:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3d0afad00 x1874671280549248/t17179869494(17179869494) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827204.595239:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827204.595245:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aae940 x1874671280549888/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827204.595259:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aae940 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280549888:192.168.203.124@tcp:35:cp.0 00000100:00100000:0.0:1787827204.595294:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ca80 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.596189:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aae940 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280549888:192.168.203.124@tcp:35:cp.0 00000002:00100000:0.0:1787827204.596209:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3d0afad00 x1874671280549248/t17179869494(17179869494) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.596267:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.596282:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.596298:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.596309:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.596317:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.596323:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.596326:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.596332:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.596336:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.596344:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.596356:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.596363:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.596369:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.596373:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.596377:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.596381:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-lfsck.sh, flags=8 00000080:00200008:0.0:1787827204.596389:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-lfsck.sh:[0x240001b71:0x3d:0x0](ffff8ab3f123db88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.596394:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x3d:0x0](ffff8ab3f123db88),name=sanity-lfsck.sh 00000002:00010000:0.0:1787827204.596406:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x3d:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.596416:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.596425:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.596433:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.596438:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x3d:0x0](ffff8ab3f123db88) 00000080:00010000:0.0:1787827204.596441:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x3d:0x0](ffff8ab3f123db88) for lock 0x268e1a602cc2344 00000080:00010000:0.0:1787827204.596446:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2344 from it ffffb86e87e83cb8 00010000:00010000:0.0:1787827204.596449:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827204.596462:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827204.596474:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.596481:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x3d:0x0] 00000008:00010000:0.0:1787827204.596490:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.596492:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7f0b0, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.596501:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.596508:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1787827204.596516:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1787827204.596519:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.596525:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b8c0ab27/0x268e1a602cc2352 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da270 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1787827204.596530:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1787827204.596535:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x3d:0x0] updating i_size 215293 i_blocks 0 00000020:00010000:0.0:1787827204.596538:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.596539:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.596541:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7f0b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.596543:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.596546:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7f0b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.596550:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3d:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827204.596589:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.596595:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.596600:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.596606:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.596611:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.596613:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.596615:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.596618:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.596620:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.596625:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.596630:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.596632:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.596635:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.596636:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.596638:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.596649:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-lnet.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:0.0:1787827204.596653:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-lnet.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.596660:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-lnet.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.596685:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96600/0x268e1a602cc2359 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.596690:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96600/0x268e1a602cc2359 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.596698:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96600/0x268e1a602cc2359 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.596707:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aaf0c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280550016:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.596745:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ca80 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.597687:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aaf0c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280550016:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.597699:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96600/0x268e1a602cc2359 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.597711:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96600/0x268e1a602cc2359 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.597719:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96600/0x268e1a602cc2359 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.597727:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96600/0x268e1a602cc2359 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.597735:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96600/0x268e1a602cc2359 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.597742:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6a96600 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.597747:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96600/0x268e1a602cc2359 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.597760:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96600/0x268e1a602cc2359 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.597768:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96600/0x268e1a602cc2359 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.597775:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.597781:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96600/0x268e1a602cc2359 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.597789:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96600/0x268e1a602cc2359 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.597804:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0aaf0c0 x1874671280550016/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.597817:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.597831:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.597862:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.597870:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.597877:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.597916:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.597925:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.597930:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.597933:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.597938:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:0.0:1787827204.597943:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.597952:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.597962:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.597967:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.597971:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.597974:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.597977:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.597986:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=sanity-lnet.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:0.0:1787827204.597989:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-lnet.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:0.0:1787827204.597997:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-lnet.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1787827204.598017:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96a00/0x268e1a602cc2360 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.598025:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96a00/0x268e1a602cc2360 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.598033:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96a00/0x268e1a602cc2360 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.598046:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aaf0c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280550144:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.598081:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ca80 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.599022:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aaf0c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280550144:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.599033:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96a00/0x268e1a602cc2360 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.599047:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96a00/0x268e1a602cc2360 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.599054:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96a00/0x268e1a602cc2360 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.599062:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96a00/0x268e1a602cc2360 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.599069:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96a00/0x268e1a602cc2360 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.599075:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6a96a00 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:0.0:1787827204.599079:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96a00/0x268e1a602cc2360 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.599086:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96a00/0x268e1a602cc2360 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.599093:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96a00/0x268e1a602cc2360 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.599113:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827204.599117:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96a00/0x268e1a602cc2360 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.599125:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a96a00/0x268e1a602cc2360 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.599138:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3d0aaf0c0 x1874671280550144/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827204.599156:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827204.599166:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:0.0:1787827204.599227:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.599236:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.599243:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.599254:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.599262:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.599267:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.599270:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.599274:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:0.0:1787827204.599278:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.599287:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.599297:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.599301:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:0.0:1787827204.599305:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.599308:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:0.0:1787827204.599311:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.599323:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.599328:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=sanity-lnet.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3c2be3b00, open_flags 80c1, mode 81ed opened 0 00000080:00200000:0.0:1787827204.599337:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=sanity-lnet.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:0.0:1787827204.599352:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: sanity-lnet.sh,[0x240001b71:0x3e:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:0.0:1787827204.599379:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.599388:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827204.599396:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827204.599403:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aaf0c0 x1874671280550400/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827204.599413:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.599426:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aaf0c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280550400:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.599545:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4ca80 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.600968:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aaf0c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280550400:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.600981:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.600999:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.601008:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.601020:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.601038:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.601049:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.601058:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.601067:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827204.601077:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3d0aaf0c0 x1874671280550400/t17179869496(17179869496) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827204.601090:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0aaf0c0 x1874671280550400/t17179869496(17179869496) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827204.601118:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827204.601130:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.601139:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/62 (0 0) 00000080:00200000:0.0:1787827204.601143:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827204.601151:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=sanity-lnet.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:0.0:1787827204.601177:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x3e:0x0](ffff8ab3f1238908) 00000080:00200000:0.0:1787827204.601279:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x3e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1787827204.601286:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x3e:0x0](ffff8ab3f1238908) 00000080:00010000:0.0:1787827204.601290:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x3e:0x0](ffff8ab3f1238908) for lock 0x268e1a602cc2367 00000080:00010000:0.0:1787827204.601298:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x3e:0x0](ffff8ab3f1238908) 00000080:00200008:0.0:1787827204.601307:0:48442:0:(file.c:1090:ll_file_open()) START file sanity-lnet.sh:[0x240001b71:0x3e:0x0](ffff8ab3f1238908), flags 100301 00000002:00100000:0.0:1787827204.601326:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0aaf0c0 x1874671280550400/t17179869496(17179869496) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827204.601364:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2367 from it ffff8ab3d0a11300 00010000:00010000:0.0:1787827204.601369:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.601379:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.601396:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x3e:0x0](ffff8ab3f1238908), inode mode 81ed mask 40 00000080:00200008:0.0:1787827204.601410:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sanity-lnet.sh:[0x240001b71:0x3e:0x0](ffff8ab3f1238908), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827204.601415:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x3e:0x0](ffff8ab3f1238908),name=sanity-lnet.sh 00000002:00010000:0.0:1787827204.601425:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x3e:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.601433:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.601443:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.601452:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.601456:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x3e:0x0](ffff8ab3f1238908) 00000080:00010000:0.0:1787827204.601459:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x3e:0x0](ffff8ab3f1238908) for lock 0x268e1a602cc2367 00000080:00010000:0.0:1787827204.601464:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2367 from it ffffb86e87e83cd8 00010000:00010000:0.0:1787827204.601467:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.601476:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827204.601490:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1787827204.601499:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x3e:0x0] 00000008:00010000:0.0:1787827204.601514:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1787827204.601516:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7fb30, osc lock ffff8ab3c6d3dd40, flags 0x400001002 00010000:00010000:0.0:1787827204.601547:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c884574e/0x268e1a602cc236e lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0xa0:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.601558:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c884574e/0x268e1a602cc236e lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0xa0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.601588:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1787827204.602581:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x3e:0x0] updating i_size 0 i_blocks 0 00000020:00010000:0.0:1787827204.602592:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:0.0:1787827204.602598:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.602601:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0b7fb30 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1787827204.602607:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:0.0:1787827204.602611:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0b7fb30 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:0.0:1787827204.602621:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1787827204.602768:0:48442:0:(file.c:2652:do_file_write_iter()) START file sanity-lnet.sh:[0x240001b71:0x3e:0x0], ppos: 0, count: 199288 00000080:00200000:0.0:1787827204.602777:0:48442:0:(file.c:2090:ll_file_io_generic()) sanity-lnet.sh: write ppos: 0, bytes: 199288 00000080:00200000:0.0:1787827204.602783:0:48442:0:(file.c:1848:ll_io_set_mirror()) sanity-lnet.sh: desiginated mirror: 0 00000020:00200000:0.0:1787827204.602813:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@0000000017c55e5a[0x110, 1, [0x240001b71:0x3e:0x0] hash] 00000020:00200000:0.0:1787827204.602816:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 199288) 0 0 00000080:00200000:0.0:1787827204.602819:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3e:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.602827:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 48] 00020000:00200000:0.0:1787827204.602835:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 199288) 199288, 199288 00020000:00200000:0.0:1787827204.602839:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1787827204.602853:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 48] 00000008:00010000:0.0:1787827204.602862:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1787827204.602866:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d0b7fb30, osc lock ffff8ab3c6d3dd40, flags 0x0 00010000:00010000:0.0:1787827204.602873:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d2d3cc00 type 11 mode 2 res 11811161088/160 (0 200703) 00010000:00010000:0.0:1787827204.602912:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ddb9ce2e/0x268e1a602cc2375 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0xa0:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.602923:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ddb9ce2e/0x268e1a602cc2375 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0xa0:0x0].0x0 rrc: 3 type: EXT [0->200703] (req 0->200703) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.602932:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ddb9ce2e/0x268e1a602cc2375 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0xa0:0x0].0x0 rrc: 3 type: EXT [0->200703] (req 0->200703) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.602947:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aade00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280550656:192.168.203.124@tcp:101:cp.0 00000100:00100000:0.0:1787827204.602981:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4cf00 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.603836:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.604067:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.604077:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c884574e/0x268e1a602cc236e lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0xa0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da2c4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.604089:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6a97a00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.604109:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c884574e/0x268e1a602cc236e lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0xa0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da2c4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.604126:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.604176:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.604223:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c884574e/0x268e1a602cc236e lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0xa0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da2c4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.604233:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.604259:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c884574e/0x268e1a602cc236e lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0xa0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da2c4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.604269:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c884574e/0x268e1a602cc236e lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0xa0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da2c4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.604288:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.604292:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.604297:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.604673:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aade00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280550656:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.604687:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ddb9ce2e/0x268e1a602cc2375 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0xa0:0x0].0x0 rrc: 2 type: EXT [0->200703] (req 0->200703) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da2cb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.604699:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ddb9ce2e/0x268e1a602cc2375 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0xa0:0x0].0x0 rrc: 2 type: EXT [0->200703] (req 0->200703) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da2cb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.604710:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ddb9ce2e/0x268e1a602cc2375 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0xa0:0x0].0x0 rrc: 2 type: EXT [0->200703] (req 0->200703) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da2cb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.605124:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.615313:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.615511:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.615521:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000088bc067/0x268e1a602cc2391 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0xa0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da31f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.615540:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f2858c00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.615567:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000088bc067/0x268e1a602cc2391 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0xa0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da31f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.615585:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.615634:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.615669:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000088bc067/0x268e1a602cc2391 lrc: 2/0,0 mode: --/PR res: [0x280000400:0xa0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da31f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.615678:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.615703:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000088bc067/0x268e1a602cc2391 lrc: 1/0,0 mode: --/PR res: [0x280000400:0xa0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da31f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.615713:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000088bc067/0x268e1a602cc2391 lrc: 0/0,0 mode: --/PR res: [0x280000400:0xa0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da31f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.615738:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.615742:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.615745:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.616533:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.625379:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.625552:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.625560:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc23b4 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0xa1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da37a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.625569:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f2858000 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.625575:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc23b4 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0xa1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da37a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.625586:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x40:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.625593:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x40:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.625651:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc23b4 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0xa1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da37a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.625657:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.625680:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc23b4 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0xa1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da37a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.625689:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc23b4 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0xa1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da37a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.625706:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.625710:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.625712:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.626542:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.637756:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.637939:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.637948:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000007604498c/0x268e1a602cc23d7 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0xa1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da3d5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.637959:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f2858400 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.637966:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000007604498c/0x268e1a602cc23d7 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0xa1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da3d5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.637985:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x41:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.637991:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x41:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.638055:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000007604498c/0x268e1a602cc23d7 lrc: 2/0,0 mode: --/PR res: [0x280000400:0xa1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da3d5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.638065:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.638091:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000007604498c/0x268e1a602cc23d7 lrc: 1/0,0 mode: --/PR res: [0x280000400:0xa1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da3d5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.638115:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000007604498c/0x268e1a602cc23d7 lrc: 0/0,0 mode: --/PR res: [0x280000400:0xa1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da3d5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.638130:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.638133:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.638135:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.638935:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:0.0:1787827204.647584:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e3b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280555136:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827204.649738:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.649965:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.649975:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d05bade1/0x268e1a602cc23fa lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0xa2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da430 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.649988:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f2858e00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.649993:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d05bade1/0x268e1a602cc23fa lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0xa2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da430 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.650007:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x42:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.650015:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x42:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.650044:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d05bade1/0x268e1a602cc23fa lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0xa2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da430 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.650050:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.650137:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d05bade1/0x268e1a602cc23fa lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0xa2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da430 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.650147:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d05bade1/0x268e1a602cc23fa lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0xa2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da430 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.650161:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.650163:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.650165:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.651160:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.661225:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.661447:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.661457:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc241d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0xa2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da48b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.661469:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f2858000 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.661476:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc241d lrc: 3/0,0 mode: PR/PR res: [0x280000400:0xa2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da48b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.661534:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x43:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.661543:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x43:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.661573:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc241d lrc: 2/0,0 mode: --/PR res: [0x280000400:0xa2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da48b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.661588:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.661616:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc241d lrc: 1/0,0 mode: --/PR res: [0x280000400:0xa2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da48b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.661627:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000006784d61/0x268e1a602cc241d lrc: 0/0,0 mode: --/PR res: [0x280000400:0xa2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da48b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.661644:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.661648:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.661652:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.662536:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:0.0:1787827204.671792:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c6e3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280557440:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827204.673756:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.673977:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.673985:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d05bade1/0x268e1a602cc2440 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0xa3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da4e6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.673994:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3f2858e00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.673998:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d05bade1/0x268e1a602cc2440 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0xa3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da4e6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.674011:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x44:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.674017:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x44:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.674040:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d05bade1/0x268e1a602cc2440 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0xa3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da4e6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.674046:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.674070:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d05bade1/0x268e1a602cc2440 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0xa3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da4e6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.674077:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d05bade1/0x268e1a602cc2440 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0xa3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da4e6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.674087:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.674089:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.674092:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.675008:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.685232:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.685450:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.685458:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000023b866ae/0x268e1a602cc2463 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0xa3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da541 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.685467:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0971400 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.685472:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000023b866ae/0x268e1a602cc2463 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0xa3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da541 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.685486:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x45:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.685494:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x45:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.685553:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000023b866ae/0x268e1a602cc2463 lrc: 2/0,0 mode: --/PR res: [0x280000400:0xa3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da541 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.685561:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.685580:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000023b866ae/0x268e1a602cc2463 lrc: 1/0,0 mode: --/PR res: [0x280000400:0xa3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da541 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.685591:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000023b866ae/0x268e1a602cc2463 lrc: 0/0,0 mode: --/PR res: [0x280000400:0xa3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da541 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.685603:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.685604:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.685607:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.686404:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:0.0:1787827204.699235:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07303c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280559744:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827204.701257:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:3.0:1787827204.701434:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.701445:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000da47bf4e/0x268e1a602cc2486 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0xa4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da59c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.701458:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0971800 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.701465:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000da47bf4e/0x268e1a602cc2486 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0xa4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da59c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.701483:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x46:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.701536:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x46:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.701568:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000da47bf4e/0x268e1a602cc2486 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0xa4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da59c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.701578:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.701605:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000da47bf4e/0x268e1a602cc2486 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0xa4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da59c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.701615:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000da47bf4e/0x268e1a602cc2486 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0xa4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da59c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.701642:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.701646:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.701650:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.702449:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.709041:0:10956:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1787827204.709056:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f07303c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280560896:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.709953:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000012edc42a/0x268e1a602cc24a9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.709969:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000012edc42a/0x268e1a602cc24a9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da5f7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.709981:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000012edc42a/0x268e1a602cc24a9 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da5f7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.709993:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000012edc42a/0x268e1a602cc24a9 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da5f7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.710005:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000012edc42a/0x268e1a602cc24a9 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da5f7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.710016:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.710028:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000012edc42a/0x268e1a602cc24a9 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da5f7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.710045:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000012edc42a/0x268e1a602cc24a9 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da5f7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.710063:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000012edc42a/0x268e1a602cc24a9 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da5f7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.710087:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000012edc42a/0x268e1a602cc24a9 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da5f7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827204.710120:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f07303c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280560896:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.712876:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.712946:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000012edc42a/0x268e1a602cc24a9 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0xa4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da5f7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.712959:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0970400 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.713007:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000012edc42a/0x268e1a602cc24a9 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0xa4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da5f7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.713031:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x47:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.713042:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x47:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.713088:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000012edc42a/0x268e1a602cc24a9 lrc: 2/0,0 mode: --/PR res: [0x280000400:0xa4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da5f7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.713110:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.713136:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000012edc42a/0x268e1a602cc24a9 lrc: 1/0,0 mode: --/PR res: [0x280000400:0xa4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da5f7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.713142:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000012edc42a/0x268e1a602cc24a9 lrc: 0/0,0 mode: --/PR res: [0x280000400:0xa4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da5f7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.713169:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.713174:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.713181:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.714124:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.715056:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba5680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280561152:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787827204.730587:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280562048:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.732828:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.732842:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000016377c9a/0x268e1a602cc24cc lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0xa5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da652 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.732858:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0970c00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.732866:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000016377c9a/0x268e1a602cc24cc lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0xa5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da652 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.732947:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x48:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.732958:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x48:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.733001:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000016377c9a/0x268e1a602cc24cc lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0xa5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da652 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.733012:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.733038:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000016377c9a/0x268e1a602cc24cc lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0xa5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da652 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.733044:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000016377c9a/0x268e1a602cc24cc lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0xa5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da652 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.733061:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.733067:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.733070:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.734012:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.740075:0:10956:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1787827204.740087:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280563200:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.740995:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dca6ddd3/0x268e1a602cc24ef lrc: 6/2,0 mode: --/PR res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.741005:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dca6ddd3/0x268e1a602cc24ef lrc: 6/2,0 mode: --/PR res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da6ad expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.741014:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dca6ddd3/0x268e1a602cc24ef lrc: 6/2,0 mode: PR/PR res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da6ad expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.741021:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dca6ddd3/0x268e1a602cc24ef lrc: 6/2,0 mode: PR/PR res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da6ad expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.741028:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dca6ddd3/0x268e1a602cc24ef lrc: 5/2,0 mode: PR/PR res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da6ad expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.741036:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.741046:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dca6ddd3/0x268e1a602cc24ef lrc: 6/2,0 mode: PR/PR res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da6ad expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.741055:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dca6ddd3/0x268e1a602cc24ef lrc: 5/1,0 mode: PR/PR res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da6ad expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.741064:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dca6ddd3/0x268e1a602cc24ef lrc: 5/1,0 mode: PR/PR res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da6ad expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.741073:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dca6ddd3/0x268e1a602cc24ef lrc: 4/0,0 mode: PR/PR res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da6ad expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827204.741086:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280563200:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.742240:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.742247:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dca6ddd3/0x268e1a602cc24ef lrc: 2/0,0 mode: PR/PR res: [0x280000400:0xa5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da6ad expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.742258:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0971200 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.742264:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dca6ddd3/0x268e1a602cc24ef lrc: 3/0,0 mode: PR/PR res: [0x280000400:0xa5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da6ad expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.742278:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x49:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.742286:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x49:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.742344:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dca6ddd3/0x268e1a602cc24ef lrc: 2/0,0 mode: --/PR res: [0x280000400:0xa5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da6ad expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.742351:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.742370:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dca6ddd3/0x268e1a602cc24ef lrc: 1/0,0 mode: --/PR res: [0x280000400:0xa5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da6ad expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.742376:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dca6ddd3/0x268e1a602cc24ef lrc: 0/0,0 mode: --/PR res: [0x280000400:0xa5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da6ad expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.742388:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.742391:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.742394:0:10955:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00010000:00010000:3.0:1787827204.742394:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.742408:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0ba7840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280563456:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1787827204.743183:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.749121:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787827204.749135:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280564352:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.749974:0:10954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b21956ef/0x268e1a602cc2512 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.749985:0:10954:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b21956ef/0x268e1a602cc2512 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da708 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.749995:0:10954:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b21956ef/0x268e1a602cc2512 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da708 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.750003:0:10954:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b21956ef/0x268e1a602cc2512 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da708 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1787827204.750016:0:10954:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b21956ef/0x268e1a602cc2512 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da708 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1787827204.750024:0:10954:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1787827204.750033:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b21956ef/0x268e1a602cc2512 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da708 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.750039:0:10954:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b21956ef/0x268e1a602cc2512 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da708 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.750045:0:10954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b21956ef/0x268e1a602cc2512 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da708 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827204.750051:0:10954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b21956ef/0x268e1a602cc2512 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da708 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827204.750061:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280564352:192.168.203.124@tcp:101:cp.0 00010000:00010000:3.0:1787827204.751206:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.751214:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b21956ef/0x268e1a602cc2512 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da708 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.751225:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0970e00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.751231:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b21956ef/0x268e1a602cc2512 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da708 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.751245:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.751252:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.751277:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b21956ef/0x268e1a602cc2512 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da708 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.751283:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.751347:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b21956ef/0x268e1a602cc2512 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da708 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.751362:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b21956ef/0x268e1a602cc2512 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da708 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.751378:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.751381:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.751383:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.752220:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:3.0:1787827204.753053:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280564608:192.168.203.124@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1787827204.754373:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000016377c9a/0x268e1a602cc2519 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da70f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.754392:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000016377c9a/0x268e1a602cc2519 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da70f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1787827204.754404:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827204.754408:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000016377c9a/0x268e1a602cc2519 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da70f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.754417:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000016377c9a/0x268e1a602cc2519 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da70f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1787827204.754428:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827204.754433:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x4a:0x0] updating i_size 24288 i_blocks 0 00000020:00010000:2.0:1787827204.754439:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:2.0:1787827204.754442:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.754446:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d068e1b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.754449:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827204.754452:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d068e1b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827204.754460:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4a:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827204.754502:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.754510:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.754519:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.754529:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.754544:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.754548:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.754551:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.754556:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:2.0:1787827204.754559:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.754569:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.754579:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.754584:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:2.0:1787827204.754588:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.754592:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1787827204.754595:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.754606:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=setup-nfs.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:2.0:1787827204.754612:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=setup-nfs.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:2.0:1787827204.754623:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: setup-nfs.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827204.754671:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68600/0x268e1a602cc2520 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.754681:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68600/0x268e1a602cc2520 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.754691:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68600/0x268e1a602cc2520 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827204.754716:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f4f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280564992:192.168.203.124@tcp:101:cp.0 00000100:00100000:2.0:1787827204.754758:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913300 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.756079:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f4f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280564992:192.168.203.124@tcp:101:cp.0 00010000:00010000:2.0:1787827204.756090:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68600/0x268e1a602cc2520 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.756112:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68600/0x268e1a602cc2520 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.756118:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68600/0x268e1a602cc2520 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.756122:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68600/0x268e1a602cc2520 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.756127:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68600/0x268e1a602cc2520 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.756131:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6d68600 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:2.0:1787827204.756134:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68600/0x268e1a602cc2520 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.756139:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68600/0x268e1a602cc2520 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.756144:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68600/0x268e1a602cc2520 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.756149:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1787827204.756152:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68600/0x268e1a602cc2520 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.756160:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68600/0x268e1a602cc2520 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827204.756170:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3c69f4f00 x1874671280564992/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787827204.756179:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1787827204.756192:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:2.0:1787827204.756219:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.756225:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.756230:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.756237:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.756242:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.756245:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.756247:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.756249:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:2.0:1787827204.756251:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.756257:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.756264:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.756267:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.756270:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.756272:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.756274:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.756280:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=setup-nfs.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=8 00000080:00200000:2.0:1787827204.756283:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=setup-nfs.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:2.0:1787827204.756288:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: setup-nfs.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827204.756305:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68a00/0x268e1a602cc2527 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.756313:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68a00/0x268e1a602cc2527 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.756321:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68a00/0x268e1a602cc2527 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827204.756336:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f4f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280565120:192.168.203.124@tcp:101:cp.0 00000100:00100000:2.0:1787827204.756374:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913300 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.757216:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f4f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280565120:192.168.203.124@tcp:101:cp.0 00010000:00010000:2.0:1787827204.757228:0:48442:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68a00/0x268e1a602cc2527 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.757236:0:48442:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68a00/0x268e1a602cc2527 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.757243:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68a00/0x268e1a602cc2527 lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.757251:0:48442:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68a00/0x268e1a602cc2527 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.757263:0:48442:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68a00/0x268e1a602cc2527 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.757270:0:48442:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6d68a00 already unused, calling callback (ffffffffc19f2990) 00010000:00010000:2.0:1787827204.757274:0:48442:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68a00/0x268e1a602cc2527 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.757281:0:48442:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68a00/0x268e1a602cc2527 lrc: 4/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.757288:0:48442:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68a00/0x268e1a602cc2527 lrc: 5/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.757293:0:48442:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1787827204.757296:0:48442:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68a00/0x268e1a602cc2527 lrc: 3/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.757303:0:48442:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68a00/0x268e1a602cc2527 lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827204.757316:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8ab3c69f4f00 x1874671280565120/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787827204.757327:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1787827204.757339:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 30 res 9663683441/1 (0 0) 00000080:00200000:2.0:1787827204.757384:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.757391:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.757396:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.757407:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.757412:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.757415:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.757417:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.757420:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83bf8 00010000:00010000:2.0:1787827204.757422:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.757427:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.757434:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.757437:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:2.0:1787827204.757439:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.757441:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1787827204.757443:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.757453:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.757457:0:48442:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=setup-nfs.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), file ffff8ab3e01f0780, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1787827204.757463:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=setup-nfs.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000002:00010000:2.0:1787827204.757475:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: setup-nfs.sh,[0x240001b71:0x4b:0x0]) in obj [0x240001b71:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1787827204.757497:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.757502:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787827204.757509:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787827204.757514:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c69f4f00 x1874671280565376/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787827204.757520:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827204.757529:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f4f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280565376:192.168.203.124@tcp:101:cp.0 00000100:00100000:2.0:1787827204.757625:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913300 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.759114:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f4f00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280565376:192.168.203.124@tcp:101:cp.0 00010000:00010000:2.0:1787827204.759125:0:48442:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.759136:0:48442:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.759143:0:48442:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.759150:0:48442:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.759158:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.759164:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.759173:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.759178:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827204.759184:0:48442:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3c69f4f00 x1874671280565376/t17179869522(17179869522) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787827204.759192:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3c69f4f00 x1874671280565376/t17179869522(17179869522) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 656/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827204.759201:0:48442:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827204.759209:0:48442:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.759215:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/75 (0 0) 00000080:00200000:2.0:1787827204.759218:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827204.759225:0:48442:0:(namei.c:1628:ll_create_it()) VFS Op:name=setup-nfs.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=open|creat 00000080:00200000:2.0:1787827204.759246:0:48442:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x4b:0x0](ffff8ab3f11fb248) 00000080:00200000:2.0:1787827204.759311:0:48442:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x4b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1787827204.759317:0:48442:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x4b:0x0](ffff8ab3f11fb248) 00000080:00010000:2.0:1787827204.759321:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x4b:0x0](ffff8ab3f11fb248) for lock 0x268e1a602cc252e 00000080:00010000:2.0:1787827204.759332:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x4b:0x0](ffff8ab3f11fb248) 00000080:00200008:2.0:1787827204.759341:0:48442:0:(file.c:1090:ll_file_open()) START file setup-nfs.sh:[0x240001b71:0x4b:0x0](ffff8ab3f11fb248), flags 100301 00000002:00100000:2.0:1787827204.759361:0:48442:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3c69f4f00 x1874671280565376/t17179869522(17179869522) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787827204.759392:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc252e from it ffff8ab3d0686120 00010000:00010000:2.0:1787827204.759396:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.759402:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827204.759414:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x4b:0x0](ffff8ab3f11fb248), inode mode 81ed mask 40 00000080:00200008:2.0:1787827204.759425:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file setup-nfs.sh:[0x240001b71:0x4b:0x0](ffff8ab3f11fb248), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827204.759429:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x4b:0x0](ffff8ab3f11fb248),name=setup-nfs.sh 00000002:00010000:2.0:1787827204.759436:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x4b:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.759442:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.759448:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827204.759453:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.759456:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x4b:0x0](ffff8ab3f11fb248) 00000080:00010000:2.0:1787827204.759458:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x4b:0x0](ffff8ab3f11fb248) for lock 0x268e1a602cc252e 00000080:00010000:2.0:1787827204.759460:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc252e from it ffffb86e87e83cd8 00010000:00010000:2.0:1787827204.759463:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.759468:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827204.759478:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4b:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827204.759484:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x4b:0x0] 00000008:00010000:2.0:1787827204.759493:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827204.759495:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d068e8b0, osc lock ffff8ab3c6ab1c08, flags 0x400001002 00010000:00010000:2.0:1787827204.759516:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000a686df1/0x268e1a602cc2535 lrc: 3/1,0 mode: --/PR res: [0x280000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.759524:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000a686df1/0x268e1a602cc2535 lrc: 3/1,0 mode: --/PR res: [0x280000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787827204.759548:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1787827204.760464:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x4b:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787827204.760469:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:2.0:1787827204.760472:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.760474:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d068e8b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.760478:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827204.760480:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d068e8b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827204.760490:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4b:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787827204.760517:0:48442:0:(file.c:2652:do_file_write_iter()) START file setup-nfs.sh:[0x240001b71:0x4b:0x0], ppos: 0, count: 1611 00000080:00200000:2.0:1787827204.760527:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 1611 bytes 00000080:00200000:2.0:1787827204.760531:0:48442:0:(file.c:2628:ll_do_tiny_write()) result: 0, original count 1611 00000080:00200000:2.0:1787827204.760535:0:48442:0:(file.c:2090:ll_file_io_generic()) setup-nfs.sh: write ppos: 0, bytes: 1611 00000080:00200000:2.0:1787827204.760539:0:48442:0:(file.c:1848:ll_io_set_mirror()) setup-nfs.sh: desiginated mirror: 0 00000020:00200000:2.0:1787827204.760544:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@00000000e4758361[0x110, 1, [0x240001b71:0x4b:0x0] hash] 00000020:00200000:2.0:1787827204.760546:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 1611) 0 0 00000080:00200000:2.0:1787827204.760548:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4b:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787827204.760554:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1787827204.760560:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1611) 1611, 1611 00020000:00200000:2.0:1787827204.760563:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827204.760574:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1787827204.760581:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1787827204.760584:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d068e8b0, osc lock ffff8ab3c6ab1c08, flags 0x0 00010000:00010000:2.0:1787827204.760590:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3c7ca6800 type 11 mode 2 res 10737419264/166 (0 4095) 00010000:00010000:2.0:1787827204.760605:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000569319e0/0x268e1a602cc253c lrc: 3/0,1 mode: --/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.760613:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000569319e0/0x268e1a602cc253c lrc: 3/0,1 mode: --/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.760618:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000569319e0/0x268e1a602cc253c lrc: 3/0,1 mode: --/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827204.760627:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f43c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280565632:192.168.203.124@tcp:101:cp.0 00000100:00100000:2.0:1787827204.760658:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913e40 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.761407:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:2.0:1787827204.761447:0:47646:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294804992 00000100:00100000:2.0:1787827204.761453:0:47646:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294804992 00000100:00100000:2.0:1787827204.761493:0:47646:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 302 00000100:00100000:2.0:1787827204.761502:0:47646:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3e0ba4b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:82330:x1874671294804992:12345-192.168.203.124@tcp:104: 00010000:00010000:2.0:1787827204.761511:0:47646:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000a686df1/0x268e1a602cc2535 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da763 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.761551:0:47646:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1787827204.761559:0:47646:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3e0ba4b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:82330:x1874671294804992:12345-192.168.203.124@tcp:104: Request processed in 58us (153us total) trans 0 rc 0/0 00000100:00100000:2.0:1787827204.761569:0:47646:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 302 00010000:00010000:3.0:1787827204.761579:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827204.761588:0:47651:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000a686df1/0x268e1a602cc2535 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da763 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.761598:0:47651:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3c6d69a00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:3.0:1787827204.761603:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000a686df1/0x268e1a602cc2535 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da763 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827204.761621:0:47651:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787827204.761628:0:47651:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827204.761686:0:47651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000a686df1/0x268e1a602cc2535 lrc: 2/0,0 mode: --/PR res: [0x280000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da763 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.761693:0:47651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827204.761716:0:47651:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000a686df1/0x268e1a602cc2535 lrc: 1/0,0 mode: --/PR res: [0x280000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da763 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.761723:0:47651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000a686df1/0x268e1a602cc2535 lrc: 0/0,0 mode: --/PR res: [0x280000400:0xa6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da763 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827204.761734:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.761737:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827204.761739:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827204.762259:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f43c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280565632:192.168.203.124@tcp:101:cp.0 00010000:00010000:2.0:1787827204.762271:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000569319e0/0x268e1a602cc253c lrc: 4/0,1 mode: --/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da76a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.762279:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000569319e0/0x268e1a602cc253c lrc: 4/0,1 mode: --/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da76a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.762286:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000569319e0/0x268e1a602cc253c lrc: 4/0,1 mode: --/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da76a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827204.762655:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:2.0:1787827204.762833:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000569319e0/0x268e1a602cc253c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da76a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.762845:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000569319e0/0x268e1a602cc253c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da76a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.762862:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000569319e0/0x268e1a602cc253c lrc: 5/0,2 mode: PW/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da76a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.762870:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000569319e0/0x268e1a602cc253c lrc: 4/0,2 mode: PW/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0da76a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.762878:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000569319e0/0x268e1a602cc253c lrc: 5/0,2 mode: PW/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da76a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.762904:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000569319e0/0x268e1a602cc253c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da76a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.762927:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) setup-nfs.sh: write [0, 1611) append 0 append lockpos 0 00000080:00200000:2.0:1787827204.762936:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 1611 bytes 00000080:00200000:2.0:1787827204.762974:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 1611, copied 1611 00000080:00200000:2.0:1787827204.762984:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000070777a53[3 0000000058769a64 2 1 000000002663f4b0 0 0 0] 00000080:00200000:2.0:1787827204.762988:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000044aea730 17ffffc0001001 3:0 ffff8ab3d0851ae0 0 lru 00000080:00200000:2.0:1787827204.762993:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000008d9d9d4f 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000f5856a27 00000000b69fec7e > 3< 0 0 > 4< 0 0 8 12288 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.762996:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000070777a53 00000080:00200000:2.0:1787827204.762997:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1787827204.763000:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 1, from 0, to 1611 prio 0 00000008:00010000:2.0:1787827204.763036:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000865266a1 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f36853e/0x268e1a602cc1089 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6bb9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827204.763047:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000865266a1 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f36853e/0x268e1a602cc1089 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6bb9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827204.763083:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f36853e/0x268e1a602cc1089 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6bb9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00100000:2.0:1787827204.763162:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3c69f43c0 - object 0x280000401:98 offset 0<>5634 00000008:00010000:2.0:1787827204.763186:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000001903ddd6 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005ad92f0c/0x268e1a602cc10cf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6c6f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827204.763194:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000001903ddd6 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005ad92f0c/0x268e1a602cc10cf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6c6f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827204.763212:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005ad92f0c/0x268e1a602cc10cf lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6c6f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827204.763236:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3c69f5a40 - object 0x280000401:99 offset 0<>1735 00000008:00010000:2.0:1787827204.763257:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000a04130da ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000205918a4/0x268e1a602cc1115 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6d25 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827204.763267:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000a04130da ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000205918a4/0x268e1a602cc1115 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6d25 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787827204.763283:0:10957:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00010000:00010000:2.0:1787827204.763298:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000205918a4/0x268e1a602cc1115 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6d25 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827204.763300:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280566016:192.168.203.124@tcp:4:cp.0 00000008:00100000:2.0:1787827204.763375:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3c69f5680 - object 0x280000401:100 offset 0<>38517 00000100:00100000:0.0:1787827204.763387:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280565760:192.168.203.124@tcp:103:ldlm_bl.0 00000008:00010000:2.0:1787827204.763403:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000002662bfd9 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000002b62c85/0x268e1a602cc115b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6ddb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787827204.763404:0:10956:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000008:00010000:2.0:1787827204.763416:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000002662bfd9 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000002b62c85/0x268e1a602cc115b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6ddb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787827204.763420:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f5680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280566144:192.168.203.124@tcp:4:cp.0 00010000:00010000:2.0:1787827204.763454:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000002b62c85/0x268e1a602cc115b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6ddb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827204.763511:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3c69f4000 - object 0x280000401:101 offset 0<>15476 00000008:00010000:2.0:1787827204.763533:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000008c4e1448/0x268e1a602cc11a1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6e91 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787827204.763538:0:10955:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:2.0:1787827204.763542:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000005e9ef40 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000008c4e1448/0x268e1a602cc11a1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6e91 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787827204.763553:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280566272:192.168.203.124@tcp:4:cp.0 00010000:00010000:2.0:1787827204.763574:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000008c4e1448/0x268e1a602cc11a1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6e91 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827204.763630:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3c69f4b40 - object 0x280000401:102 offset 0<>9198 00000008:00010000:2.0:1787827204.763659:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000ff57aff0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000004d53dce0/0x268e1a602cc11e7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6f47 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787827204.763665:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280566400:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827204.763676:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000ff57aff0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000004d53dce0/0x268e1a602cc11e7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6f47 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827204.763703:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000004d53dce0/0x268e1a602cc11e7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6f47 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827204.763739:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3c69f70c0 - object 0x280000401:103 offset 0<>4207 00000008:00010000:2.0:1787827204.763760:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000005855d0cb ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ca914e3d/0x268e1a602cc122d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6ffd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827204.763769:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000005855d0cb ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ca914e3d/0x268e1a602cc122d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6ffd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827204.763788:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ca914e3d/0x268e1a602cc122d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6ffd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827204.763794:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280566528:192.168.203.124@tcp:4:cp.0 00000008:00100000:2.0:1787827204.763817:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3c69f4780 - object 0x280000401:104 offset 0<>704 00000008:00010000:2.0:1787827204.763834:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000006bac1124 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000cc942554/0x268e1a602cc1273 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d70b3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827204.763842:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000006bac1124 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000cc942554/0x268e1a602cc1273 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d70b3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787827204.763845:0:10956:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:0.0:1787827204.763859:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280566656:192.168.203.124@tcp:4:cp.0 00010000:00010000:2.0:1787827204.763871:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000cc942554/0x268e1a602cc1273 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d70b3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827204.763960:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3c69f6580 - object 0x280000401:105 offset 0<>4915 00000008:00010000:2.0:1787827204.763986:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000085946cf0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d2f60288/0x268e1a602cc12b9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7169 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827204.763995:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000085946cf0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d2f60288/0x268e1a602cc12b9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7169 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827204.764031:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d2f60288/0x268e1a602cc12b9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7169 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827204.764085:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3c69f7480 - object 0x280000401:106 offset 0<>16105 00000008:00010000:2.0:1787827204.765716:0:10954:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000865266a1 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f36853e/0x268e1a602cc1089 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6bb9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00010000:2.0:1787827204.765763:0:10954:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000865266a1 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f36853e/0x268e1a602cc1089 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6bb9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00010000:3.0:1787827204.765765:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000002662bfd9 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000002b62c85/0x268e1a602cc115b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6ddb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827204.765800:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000ff57aff0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000004d53dce0/0x268e1a602cc11e7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6f47 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827204.765812:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280565888:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827204.765830:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000ff57aff0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000004d53dce0/0x268e1a602cc11e7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6f47 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827204.765873:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280566528:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827204.765878:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000002662bfd9 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000002b62c85/0x268e1a602cc115b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6ddb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1787827204.765922:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280566272:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827204.767127:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000001903ddd6 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005ad92f0c/0x268e1a602cc10cf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6c6f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827204.767148:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000001903ddd6 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005ad92f0c/0x268e1a602cc10cf lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6c6f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827204.767150:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000005855d0cb ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ca914e3d/0x268e1a602cc122d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6ffd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827204.767175:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280566016:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827204.767191:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000005855d0cb ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ca914e3d/0x268e1a602cc122d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6ffd expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827204.767229:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280566656:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827204.767235:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000005e9ef40 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000008c4e1448/0x268e1a602cc11a1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6e91 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1787827204.767320:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000005e9ef40 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000008c4e1448/0x268e1a602cc11a1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6e91 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1787827204.767338:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280566400:192.168.203.124@tcp:4:cp.0 00000100:00100000:2.0:1787827204.768004:0:10956:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8ab3c584d000: committing for last_committed 4294967427 gen 1 00000100:00100000:2.0:1787827204.768047:0:10956:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8ab3c69f5a40 x1874671280566016/t4294967428(4294967428) o4->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 2224/448 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1787827204.768084:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000a04130da ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000205918a4/0x268e1a602cc1115 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6d25 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827204.768120:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000a04130da ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000205918a4/0x268e1a602cc1115 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6d25 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827204.768140:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f5680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280566144:192.168.203.124@tcp:4:cp.0 00000100:00100000:2.0:1787827204.773281:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280567808:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.774347:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.774362:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da7be expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.774379:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da7be expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.774388:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da7be expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.774399:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da7be expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.774407:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787827204.774416:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da7be expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.774422:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da7be expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.774428:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da7be expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.774434:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da7be expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787827204.774444:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280567808:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827204.775600:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:2.0:1787827204.775652:0:47646:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294805248 00000100:00100000:2.0:1787827204.775660:0:47646:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294805248 00000100:00100000:2.0:1787827204.775720:0:47646:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 304 00000100:00100000:2.0:1787827204.775733:0:47646:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3e0ba6d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:77209:x1874671294805248:12345-192.168.203.124@tcp:104: 00010000:00010000:2.0:1787827204.775747:0:47646:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da7be expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.775802:0:47646:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1787827204.775809:0:47646:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3e0ba6d00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:77209:x1874671294805248:12345-192.168.203.124@tcp:104: Request processed in 80us (213us total) trans 0 rc 0/0 00000100:00100000:2.0:1787827204.775817:0:47646:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 304 00010000:00010000:0.0:1787827204.775838:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787827204.775846:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da7be expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.775856:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0855800 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:0.0:1787827204.775861:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da7be expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.775875:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.775883:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787827204.775947:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da7be expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.775958:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787827204.775982:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da7be expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827204.775986:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aada40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280568064:192.168.203.124@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1787827204.775997:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d30b4757/0x268e1a602cc2558 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0xa7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da7be expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.776014:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827204.776018:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827204.776021:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827204.776846:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:2.0:1787827204.777091:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a85cda38/0x268e1a602cc255f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da7c5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.777125:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a85cda38/0x268e1a602cc255f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da7c5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.777143:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a85cda38/0x268e1a602cc255f lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da7c5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.777157:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a85cda38/0x268e1a602cc255f lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0da7c5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.777170:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a85cda38/0x268e1a602cc255f lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da7c5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.777179:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a85cda38/0x268e1a602cc255f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da7c5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.777213:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) sgpdd-survey.sh: write [0, 1765) append 0 append lockpos 0 00000080:00200000:2.0:1787827204.777228:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 1765 bytes 00000080:00200000:2.0:1787827204.777281:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 1765, copied 1765 00000080:00200000:2.0:1787827204.777295:0:48442:0:(rw26.c:933:ll_write_end()) page@000000009d100499[3 000000008a650caf 2 1 000000002663f4b0 0 0 0] 00000080:00200000:2.0:1787827204.777300:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000cff10c9e 17ffffc0001001 3:0 ffff8ab3d0851bc8 0 lru 00000080:00200000:2.0:1787827204.777308:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000017aad16a 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000006a588f83 000000005ee851eb > 3< 0 0 > 4< 0 0 8 2453504 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.777313:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000009d100499 00000080:00200000:2.0:1787827204.777315:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1787827204.777320:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 1, from 0, to 1765 prio 0 00000008:00010000:2.0:1787827204.777342:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 0000000037c1467f ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a85cda38/0x268e1a602cc255f lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da7c5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.777365:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f13f12f8, count 1, dirtied 1 00000080:00200000:2.0:1787827204.777371:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 1 pages 1765 bytes, tot: 1765 00000080:00200000:2.0:1787827204.777374:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) sgpdd-survey.sh: write bytes 0, result: 1765 00000020:00010000:2.0:1787827204.777392:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:2.0:1787827204.777396:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.777399:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3c3760e30 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1787827204.777405:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a85cda38/0x268e1a602cc255f lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da7c5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.777416:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a85cda38/0x268e1a602cc255f lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da7c5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1787827204.777429:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d0681308 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827204.777433:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3c3760e30 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827204.777441:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 1765 bytes 00000080:00200000:2.0:1787827204.777448:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1787827204.777455:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4c:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827204.777460:0:48442:0:(file.c:2240:ll_file_io_generic()) sgpdd-survey.sh: 2 io complete with rc: 0, result: 1765, restart: 0 00000080:00200000:2.0:1787827204.777467:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x4c:0x0] updating i_size 1765 i_blocks 0 00000080:00200000:2.0:1787827204.777472:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 1765 00000080:00200008:2.0:1787827204.777502:0:48442:0:(file.c:479:ll_file_release()) START file sgpdd-survey.sh:[0x240001b71:0x4c:0x0](ffff8ab3f13f1148), flags 100001 00000002:00100000:2.0:1787827204.777557:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3d22fe940 x1874671280567680/t17179869524(17179869524) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787827204.777577:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787827204.777584:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c69f70c0 x1874671280568320/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787827204.777604:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f70c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280568320:192.168.203.124@tcp:35:cp.0 00000100:00100000:2.0:1787827204.777651:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913a80 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.777803:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aada40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280568064:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787827204.778666:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f70c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280568320:192.168.203.124@tcp:35:cp.0 00000002:00100000:2.0:1787827204.778682:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3d22fe940 x1874671280567680/t17179869524(17179869524) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 584/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787827204.778731:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.778743:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.778755:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.778763:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.778769:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.778774:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.778776:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.778781:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:2.0:1787827204.778784:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.778790:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.778802:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.778808:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.778813:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.778816:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.778818:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.778821:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sgpdd-survey.sh, flags=8 00000080:00200008:2.0:1787827204.778827:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file sgpdd-survey.sh:[0x240001b71:0x4c:0x0](ffff8ab3f13f1148), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827204.778831:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x4c:0x0](ffff8ab3f13f1148),name=sgpdd-survey.sh 00000002:00010000:2.0:1787827204.778839:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x4c:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.778844:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854e00/0x268e1a602cc2551 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da7b0 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827204.778850:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854e00/0x268e1a602cc2551 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da7b0 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827204.778855:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.778857:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x4c:0x0](ffff8ab3f13f1148) 00000080:00010000:2.0:1787827204.778859:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x4c:0x0](ffff8ab3f13f1148) for lock 0x268e1a602cc2551 00000080:00010000:2.0:1787827204.778861:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2551 from it ffffb86e87e83cb8 00010000:00010000:2.0:1787827204.778862:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854e00/0x268e1a602cc2551 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da7b0 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827204.778868:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854e00/0x268e1a602cc2551 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da7b0 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827204.778880:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4c:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827204.778922:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x4c:0x0] 00000008:00010000:2.0:1787827204.778937:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827204.778940:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d068e4b0, osc lock ffff8ab3c6ab1728, flags 0x400001002 00010000:00010000:2.0:1787827204.778953:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a85cda38/0x268e1a602cc255f lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0xa7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da7c5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.778962:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a85cda38/0x268e1a602cc255f lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da7c5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1787827204.778970:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827204.778973:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a85cda38/0x268e1a602cc255f lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da7c5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.778979:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a85cda38/0x268e1a602cc255f lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da7c5 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1787827204.778985:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827204.778989:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x4c:0x0] updating i_size 1765 i_blocks 0 00000020:00010000:2.0:1787827204.778993:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_release():245 00000020:00010000:2.0:1787827204.778995:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.778996:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d068e4b0 (ffff8ab3c6e737e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.778999:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d07ef3d8 (ffff8ab3c6e73998/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827204.779001:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d068e4b0 (ffff8ab3c6e73998/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827204.779009:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4c:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827204.779046:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.779053:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.779058:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.779073:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.779081:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.779085:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.779088:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.779104:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:2.0:1787827204.779109:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.779119:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.779129:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.779134:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:2.0:1787827204.779137:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.779139:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1787827204.779141:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.779151:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=test-framework.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:2.0:1787827204.779155:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=test-framework.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:2.0:1787827204.779164:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: test-framework.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827204.779191:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69200/0x268e1a602cc2566 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.779197:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69200/0x268e1a602cc2566 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.779208:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69200/0x268e1a602cc2566 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827204.779218:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f4000 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280568448:192.168.203.124@tcp:101:cp.0 00000100:00100000:2.0:1787827204.779258:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913a80 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.783852:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280568960:192.168.203.124@tcp:101:cp.0 00000080:00200000:2.0:1787827204.784859:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x4d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1787827204.784866:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3c4942018 (ffff8ab3c6ae1908/0) at cl_lock_release():245 00000020:00010000:2.0:1787827204.784869:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3c4942018 (ffff8ab3c6ae1908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.784872:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3c3760eb0 (ffff8ab3c6ae1d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.784876:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3c4942018 (ffff8ab3c6ae1908/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827204.784878:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3c3760eb0 (ffff8ab3c6ae1908/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827204.784901:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4d:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787827204.785119:0:48442:0:(file.c:2652:do_file_write_iter()) START file test-framework.sh:[0x240001b71:0x4d:0x0], ppos: 0, count: 317548 00000080:00200000:2.0:1787827204.785129:0:48442:0:(file.c:2090:ll_file_io_generic()) test-framework.sh: write ppos: 0, bytes: 317548 00000080:00200000:2.0:1787827204.785135:0:48442:0:(file.c:1848:ll_io_set_mirror()) test-framework.sh: desiginated mirror: 0 00000020:00200000:2.0:1787827204.785141:0:48442:0:(cl_io.c:218:cl_io_rw_init()) header@000000006883af57[0x110, 1, [0x240001b71:0x4d:0x0] hash] 00000020:00200000:2.0:1787827204.785143:0:48442:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 317548) 0 0 00000080:00200000:2.0:1787827204.785145:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787827204.785152:0:48442:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 77] 00020000:00200000:2.0:1787827204.785160:0:48442:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 317548) 317548, 317548 00020000:00200000:2.0:1787827204.785163:0:48442:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827204.785174:0:48442:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 77] 00000008:00010000:2.0:1787827204.785182:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1787827204.785185:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d068e4b0, osc lock ffff8ab3c6ab1728, flags 0x0 00010000:00010000:2.0:1787827204.785192:0:48442:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3c7ca6800 type 11 mode 2 res 10737419264/167 (0 319487) 00010000:00010000:2.0:1787827204.785210:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 3/0,1 mode: --/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.785221:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 3/0,1 mode: --/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 3 type: EXT [0->319487] (req 0->319487) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.785228:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 3/0,1 mode: --/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 3 type: EXT [0->319487] (req 0->319487) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827204.785239:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f43c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280569088:192.168.203.124@tcp:101:cp.0 00000100:00100000:2.0:1787827204.785269:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913840 going to sleep for 11 seconds 00000100:00100000:3.0:1787827204.786063:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:2.0:1787827204.786126:0:47646:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294805504 00000100:00100000:2.0:1787827204.786132:0:47646:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294805504 00000100:00100000:2.0:1787827204.786174:0:47646:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 306 00000100:00100000:2.0:1787827204.786182:0:47646:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3e0ba4780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:77210:x1874671294805504:12345-192.168.203.124@tcp:104: 00010000:00010000:2.0:1787827204.786192:0:47646:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000bef9735b/0x268e1a602cc257b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0xa7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da819 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.786235:0:47646:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1787827204.786241:0:47646:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3e0ba4780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:77210:x1874671294805504:12345-192.168.203.124@tcp:104: Request processed in 60us (179us total) trans 0 rc 0/0 00000100:00100000:2.0:1787827204.786247:0:47646:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 306 00000100:00100000:3.0:1787827204.786436:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280569216:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787827204.786941:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f43c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280569088:192.168.203.124@tcp:101:cp.0 00010000:00010000:2.0:1787827204.786954:0:48442:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: --/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->319487] (req 0->319487) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.786962:0:48442:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: --/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->319487] (req 0->319487) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.786974:0:48442:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: --/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->319487] (req 0->319487) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787827204.787217:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:2.0:1787827204.787396:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.787408:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.787421:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.787432:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.787444:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.787453:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.787472:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) test-framework.sh: write [0, 317548) append 0 append lockpos 0 00000080:00200000:2.0:1787827204.787487:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.787527:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1787827204.787540:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000b65d6273[3 00000000f3041499 2 1 00000000431eb59a 0 0 0] 00000080:00200000:2.0:1787827204.787545:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000490fd22a 17ffffc0001001 3:0 ffff8ab3d0851910 0 lru 00000080:00200000:2.0:1787827204.787553:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000bf9d9ebd 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.787555:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000b65d6273 00000080:00200000:2.0:1787827204.787557:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1787827204.787560:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.787621:0:48442:0:(rw26.c:875:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1787827204.787626:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000095d66ad0[3 00000000f3041499 2 1 00000000431eb59a 0 0 1] 00000080:00200000:2.0:1787827204.787630:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000e7f73fac 17ffffc0001001 3:0 ffff8ab3d0851828 1 lru 00000080:00200000:2.0:1787827204.787636:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000a6f17e72 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.787638:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000095d66ad0 00000080:00200000:2.0:1787827204.787639:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1787827204.787641:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.787658:0:48442:0:(rw26.c:875:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1787827204.787663:0:48442:0:(rw26.c:933:ll_write_end()) page@000000005996882a[3 00000000f3041499 2 1 00000000431eb59a 0 0 2] 00000080:00200000:2.0:1787827204.787667:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000a96fa6d1 17ffffc0001001 3:0 ffff8ab3d0851cb0 2 lru 00000080:00200000:2.0:1787827204.787673:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000025ee072c 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.787676:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000005996882a 00000080:00200000:2.0:1787827204.787677:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1787827204.787679:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.787702:0:48442:0:(rw26.c:875:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1787827204.787706:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000034f8489c[3 00000000f3041499 2 1 00000000431eb59a 0 0 3] 00000080:00200000:2.0:1787827204.787709:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000c6d37456 17ffffc0001001 3:0 ffff8ab3d08a9910 3 lru 00000080:00200000:2.0:1787827204.787715:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000cb0e5b7c 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.787718:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000034f8489c 00000080:00200000:2.0:1787827204.787719:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1787827204.787721:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.787735:0:48442:0:(rw26.c:875:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1787827204.787740:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000de4af928[3 00000000f3041499 2 1 00000000431eb59a 0 0 4] 00000080:00200000:2.0:1787827204.787743:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000000a4af0df 17ffffc0001001 3:0 ffff8ab3d08a9828 4 lru 00000080:00200000:2.0:1787827204.787748:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000044bce132 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.787750:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000de4af928 00000080:00200000:2.0:1787827204.787752:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1787827204.787754:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.787768:0:48442:0:(rw26.c:875:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1787827204.787772:0:48442:0:(rw26.c:933:ll_write_end()) page@000000006d5b2217[3 00000000f3041499 2 1 00000000431eb59a 0 0 5] 00000080:00200000:2.0:1787827204.787775:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000ee28fc1a 17ffffc0001001 3:0 ffff8ab3d08a9cb0 5 lru 00000080:00200000:2.0:1787827204.787781:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000d127728a 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.787784:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000006d5b2217 00000080:00200000:2.0:1787827204.787785:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1787827204.787787:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.787800:0:48442:0:(rw26.c:875:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1787827204.787805:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000ec85a0ed[3 00000000f3041499 2 1 00000000431eb59a 0 0 6] 00000080:00200000:2.0:1787827204.787808:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000004c0c33f8 17ffffc0001001 3:0 ffff8ab3d08a9d98 6 lru 00000080:00200000:2.0:1787827204.787813:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000005787cdd4 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.787816:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000ec85a0ed 00000080:00200000:2.0:1787827204.787817:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1787827204.787819:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.787836:0:48442:0:(rw26.c:875:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:2.0:1787827204.787841:0:48442:0:(rw26.c:933:ll_write_end()) page@000000003c345f14[3 00000000f3041499 2 1 00000000431eb59a 0 0 7] 00000080:00200000:2.0:1787827204.787845:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000daf8314c 17ffffc0001001 3:0 ffff8ab3d08a92b8 7 lru 00000080:00200000:2.0:1787827204.787852:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000fee687c9 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.787855:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000003c345f14 00000080:00200000:2.0:1787827204.787856:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1787827204.787858:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.787874:0:48442:0:(rw26.c:875:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:2.0:1787827204.787880:0:48442:0:(rw26.c:933:ll_write_end()) page@000000000eeab52e[3 00000000f3041499 2 1 00000000431eb59a 0 0 8] 00000080:00200000:2.0:1787827204.787883:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000024d4bdd1 17ffffc0001001 3:0 ffff8ab3d08a9488 8 lru 00000080:00200000:2.0:1787827204.787924:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000668629ad 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.787928:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000000eeab52e 00000080:00200000:2.0:1787827204.787930:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1787827204.787933:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.787953:0:48442:0:(rw26.c:875:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:2.0:1787827204.787958:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000d579d909[3 00000000f3041499 2 1 00000000431eb59a 0 0 9] 00000080:00200000:2.0:1787827204.787962:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000898c90e2 17ffffc0001001 3:0 ffff8ab3d08a93a0 9 lru 00000080:00200000:2.0:1787827204.787969:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000007aa3f178 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.787972:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000d579d909 00000080:00200000:2.0:1787827204.787974:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1787827204.787976:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000100:00100000:0.0:1787827204.787990:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280569216:192.168.203.124@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1787827204.788004:0:48442:0:(rw26.c:875:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788011:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000e9d608bd[3 00000000f3041499 2 1 00000000431eb59a 0 0 10] 00000080:00200000:2.0:1787827204.788015:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000002e88d191 17ffffc0001021 2:0 ffff8ab3d08a99f8 10 lru 00000080:00200000:2.0:1787827204.788022:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000059c0810d 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788025:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000e9d608bd 00000080:00200000:2.0:1787827204.788027:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1787827204.788029:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788046:0:48442:0:(rw26.c:875:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788051:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000040d78429[3 00000000f3041499 2 1 00000000431eb59a 0 0 11] 00000080:00200000:2.0:1787827204.788054:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000c78e465e 17ffffc0001001 3:0 ffff8ab3d08a90e8 11 lru 00000080:00200000:2.0:1787827204.788059:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000008257a4ab 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788062:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000040d78429 00000080:00200000:2.0:1787827204.788063:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1787827204.788065:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788080:0:48442:0:(rw26.c:875:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788085:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000003c72b1f[3 00000000f3041499 2 1 00000000431eb59a 0 0 12] 00000080:00200000:2.0:1787827204.788088:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000027fd07c1 17ffffc0001001 3:0 ffff8ab3d08a9740 12 lru 00000080:00200000:2.0:1787827204.788109:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000f18dd650 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788112:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000003c72b1f 00000080:00200000:2.0:1787827204.788113:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1787827204.788116:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788136:0:48442:0:(rw26.c:875:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788141:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000af773f08[3 00000000f3041499 2 1 00000000431eb59a 0 0 13] 00000080:00200000:2.0:1787827204.788144:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000e08141b4 17ffffc0001001 3:0 ffff8ab3d08a9570 13 lru 00000080:00200000:2.0:1787827204.788150:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000a7048786 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788153:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000af773f08 00000080:00200000:2.0:1787827204.788154:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1787827204.788157:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788173:0:48442:0:(rw26.c:875:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788178:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000cc6c972c[3 00000000f3041499 2 1 00000000431eb59a 0 0 14] 00000080:00200000:2.0:1787827204.788181:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000464c2444 17ffffc0001001 3:0 ffff8ab3d08a91d0 14 lru 00000080:00200000:2.0:1787827204.788187:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000009ee1dbc8 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788190:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000cc6c972c 00000080:00200000:2.0:1787827204.788191:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1787827204.788193:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788209:0:48442:0:(rw26.c:875:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788214:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000065b9eac[3 00000000f3041499 2 1 00000000431eb59a 0 0 15] 00000080:00200000:2.0:1787827204.788217:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000c74020db 17ffffc0001001 3:0 ffff8ab3d08a9658 15 lru 00000080:00200000:2.0:1787827204.788223:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c6966b80 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788226:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000065b9eac 00000080:00200000:2.0:1787827204.788227:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1787827204.788229:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788246:0:48442:0:(rw26.c:875:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788250:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000055a1689e[3 00000000f3041499 2 1 00000000431eb59a 0 0 16] 00000080:00200000:2.0:1787827204.788254:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000665a97de 17ffffc0001001 3:0 ffff8ab3d08a9000 16 lru 00000080:00200000:2.0:1787827204.788260:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000fce07cd3 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788262:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000055a1689e 00000080:00200000:2.0:1787827204.788263:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1787827204.788265:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788281:0:48442:0:(rw26.c:875:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788286:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000a65792cf[3 00000000f3041499 2 1 00000000431eb59a 0 0 17] 00000080:00200000:2.0:1787827204.788289:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000029ca7b0e 17ffffc0001001 3:0 ffff8ab3d08a9e80 17 lru 00000080:00200000:2.0:1787827204.788295:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000003b73954 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788298:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000a65792cf 00000080:00200000:2.0:1787827204.788299:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1787827204.788301:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788316:0:48442:0:(rw26.c:875:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788321:0:48442:0:(rw26.c:933:ll_write_end()) page@000000008f9b6824[3 00000000f3041499 2 1 00000000431eb59a 0 0 18] 00000080:00200000:2.0:1787827204.788324:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000d3098b0b 17ffffc0001001 3:0 ffff8ab3d08a9ae0 18 lru 00000080:00200000:2.0:1787827204.788330:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000061b39b8a 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788333:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000008f9b6824 00000080:00200000:2.0:1787827204.788334:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1787827204.788336:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788355:0:48442:0:(rw26.c:875:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788360:0:48442:0:(rw26.c:933:ll_write_end()) page@000000000787a59f[3 00000000f3041499 2 1 00000000431eb59a 0 0 19] 00000080:00200000:2.0:1787827204.788363:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000005a92aa5e 17ffffc0001001 3:0 ffff8ab3d08a9bc8 19 lru 00000080:00200000:2.0:1787827204.788369:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000068789079 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788372:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000000787a59f 00000080:00200000:2.0:1787827204.788374:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 20. 00000080:00200000:2.0:1787827204.788376:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788402:0:48442:0:(rw26.c:875:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788407:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000085a87558[3 00000000f3041499 2 1 00000000431eb59a 0 0 20] 00000080:00200000:2.0:1787827204.788410:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000009038ee99 17ffffc0001001 3:0 ffff8ab3d06143a0 20 lru 00000080:00200000:2.0:1787827204.788416:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000000e97fcdc 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788418:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000085a87558 00000080:00200000:2.0:1787827204.788419:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1787827204.788420:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788433:0:48442:0:(rw26.c:875:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788436:0:48442:0:(rw26.c:933:ll_write_end()) page@000000006e246a46[3 00000000f3041499 2 1 00000000431eb59a 0 0 21] 00000080:00200000:2.0:1787827204.788438:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000deb0c084 17ffffc0001001 3:0 ffff8ab3d06149f8 21 lru 00000080:00200000:2.0:1787827204.788442:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000a2d046a6 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788443:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000006e246a46 00000080:00200000:2.0:1787827204.788444:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 22. 00000080:00200000:2.0:1787827204.788445:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788454:0:48442:0:(rw26.c:875:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788457:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000057ce27c4[3 00000000f3041499 2 1 00000000431eb59a 0 0 22] 00000080:00200000:2.0:1787827204.788458:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000000c37ad59 17ffffc0001001 3:0 ffff8ab3d06140e8 22 lru 00000080:00200000:2.0:1787827204.788462:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e9ba7200 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788463:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000057ce27c4 00000080:00200000:2.0:1787827204.788464:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 23. 00000080:00200000:2.0:1787827204.788465:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788474:0:48442:0:(rw26.c:875:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788477:0:48442:0:(rw26.c:933:ll_write_end()) page@000000002a6c52b0[3 00000000f3041499 2 1 00000000431eb59a 0 0 23] 00000080:00200000:2.0:1787827204.788479:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000f5ae4b33 17ffffc0001001 3:0 ffff8ab3d0614740 23 lru 00000080:00200000:2.0:1787827204.788483:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000005ba24467 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788485:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000002a6c52b0 00000080:00200000:2.0:1787827204.788486:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 24. 00000080:00200000:2.0:1787827204.788487:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788496:0:48442:0:(rw26.c:875:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788498:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000738ad6c1[3 00000000f3041499 2 1 00000000431eb59a 0 0 24] 00000080:00200000:2.0:1787827204.788500:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000004bc30be8 17ffffc0001001 3:0 ffff8ab3d0614570 24 lru 00000080:00200000:2.0:1787827204.788503:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ece1cb05 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788504:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000738ad6c1 00000080:00200000:2.0:1787827204.788505:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 25. 00000080:00200000:2.0:1787827204.788506:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788517:0:48442:0:(rw26.c:875:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788520:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000063b90d20[3 00000000f3041499 2 1 00000000431eb59a 0 0 25] 00000080:00200000:2.0:1787827204.788521:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000033e4d926 17ffffc0001021 2:0 ffff8ab3d06141d0 25 lru 00000080:00200000:2.0:1787827204.788525:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000043464d3e 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788526:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000063b90d20 00000080:00200000:2.0:1787827204.788527:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 26. 00000080:00200000:2.0:1787827204.788528:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788536:0:48442:0:(rw26.c:875:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788539:0:48442:0:(rw26.c:933:ll_write_end()) page@000000005eb79192[3 00000000f3041499 2 1 00000000431eb59a 0 0 26] 00000080:00200000:2.0:1787827204.788540:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000269e6e30 17ffffc0001001 3:0 ffff8ab3d0614658 26 lru 00000080:00200000:2.0:1787827204.788544:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c08459ce 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788546:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000005eb79192 00000080:00200000:2.0:1787827204.788547:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 27. 00000080:00200000:2.0:1787827204.788548:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788557:0:48442:0:(rw26.c:875:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788560:0:48442:0:(rw26.c:933:ll_write_end()) page@000000007e5bfe3e[3 00000000f3041499 2 1 00000000431eb59a 0 0 27] 00000080:00200000:2.0:1787827204.788562:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000001dd97a0f 17ffffc0001001 3:0 ffff8ab3d0614000 27 lru 00000080:00200000:2.0:1787827204.788566:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000927bebb6 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788568:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000007e5bfe3e 00000080:00200000:2.0:1787827204.788568:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 28. 00000080:00200000:2.0:1787827204.788570:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788578:0:48442:0:(rw26.c:875:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788581:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000069231eb8[3 00000000f3041499 2 1 00000000431eb59a 0 0 28] 00000080:00200000:2.0:1787827204.788583:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000cf271fd7 17ffffc0001001 3:0 ffff8ab3d0614e80 28 lru 00000080:00200000:2.0:1787827204.788586:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000d9d24090 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788588:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000069231eb8 00000080:00200000:2.0:1787827204.788589:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 29. 00000080:00200000:2.0:1787827204.788590:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788598:0:48442:0:(rw26.c:875:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788601:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000259aaea3[3 00000000f3041499 2 1 00000000431eb59a 0 0 29] 00000080:00200000:2.0:1787827204.788602:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000099363d22 17ffffc0001001 3:0 ffff8ab3d0614ae0 29 lru 00000080:00200000:2.0:1787827204.788606:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ed3377e2 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788607:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000259aaea3 00000080:00200000:2.0:1787827204.788608:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 30. 00000080:00200000:2.0:1787827204.788609:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788617:0:48442:0:(rw26.c:875:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788619:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000539c9e7b[3 00000000f3041499 2 1 00000000431eb59a 0 0 30] 00000080:00200000:2.0:1787827204.788621:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000ac16242d 17ffffc0001001 3:0 ffff8ab3d0614bc8 30 lru 00000080:00200000:2.0:1787827204.788625:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000000d08b2b3 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788626:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000539c9e7b 00000080:00200000:2.0:1787827204.788626:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 31. 00000080:00200000:2.0:1787827204.788627:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788635:0:48442:0:(rw26.c:875:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788638:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000fdb53b8c[3 00000000f3041499 2 1 00000000431eb59a 0 0 31] 00000080:00200000:2.0:1787827204.788640:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000016f110c6 17ffffc0001001 3:0 ffff8ab3d0614910 31 lru 00000080:00200000:2.0:1787827204.788643:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ce744f3f 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788645:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000fdb53b8c 00000080:00200000:2.0:1787827204.788646:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 32. 00000080:00200000:2.0:1787827204.788647:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788656:0:48442:0:(rw26.c:875:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788659:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000066f1a14a[3 00000000f3041499 2 1 00000000431eb59a 0 0 32] 00000080:00200000:2.0:1787827204.788661:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000343c241f 17ffffc0001001 3:0 ffff8ab3d0614828 32 lru 00000080:00200000:2.0:1787827204.788665:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000007dba71a3 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788666:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000066f1a14a 00000080:00200000:2.0:1787827204.788667:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 33. 00000080:00200000:2.0:1787827204.788668:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788676:0:48442:0:(rw26.c:875:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788679:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000017a7d421[3 00000000f3041499 2 1 00000000431eb59a 0 0 33] 00000080:00200000:2.0:1787827204.788680:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000086f03ed3 17ffffc0001001 3:0 ffff8ab3d0614cb0 33 lru 00000080:00200000:2.0:1787827204.788684:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000009cb6e03d 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788685:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000017a7d421 00000080:00200000:2.0:1787827204.788686:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 34. 00000080:00200000:2.0:1787827204.788687:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 34 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788697:0:48442:0:(rw26.c:875:ll_write_end()) pos 139264, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788700:0:48442:0:(rw26.c:933:ll_write_end()) page@000000000fa8ad1b[3 00000000f3041499 2 1 00000000431eb59a 0 0 34] 00000080:00200000:2.0:1787827204.788702:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000cee5374f 17ffffc0001001 3:0 ffff8ab3d0614d98 34 lru 00000080:00200000:2.0:1787827204.788706:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000095370771 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788708:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000000fa8ad1b 00000080:00200000:2.0:1787827204.788709:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 35. 00000080:00200000:2.0:1787827204.788710:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 35 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788718:0:48442:0:(rw26.c:875:ll_write_end()) pos 143360, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788720:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000d1e55bf0[3 00000000f3041499 2 1 00000000431eb59a 0 0 35] 00000080:00200000:2.0:1787827204.788722:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000005cf60bc5 17ffffc0001001 3:0 ffff8ab3d06142b8 35 lru 00000080:00200000:2.0:1787827204.788725:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c8f2971d 35: 1< 0 - - > 2< 143360 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788727:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000d1e55bf0 00000080:00200000:2.0:1787827204.788728:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 36. 00000080:00200000:2.0:1787827204.788729:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 36 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788737:0:48442:0:(rw26.c:875:ll_write_end()) pos 147456, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788740:0:48442:0:(rw26.c:933:ll_write_end()) page@000000003ac20f98[3 00000000f3041499 2 1 00000000431eb59a 0 0 36] 00000080:00200000:2.0:1787827204.788743:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000009a5f8a05 17ffffc0001001 3:0 ffff8ab3d0614488 36 lru 00000080:00200000:2.0:1787827204.788746:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000a8c8a246 36: 1< 0 - - > 2< 147456 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788748:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000003ac20f98 00000080:00200000:2.0:1787827204.788749:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 37. 00000080:00200000:2.0:1787827204.788750:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 37 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788767:0:48442:0:(rw26.c:875:ll_write_end()) pos 151552, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788772:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000069359959[3 00000000f3041499 2 1 00000000431eb59a 0 0 37] 00000080:00200000:2.0:1787827204.788775:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000008f253025 17ffffc0001001 3:0 ffff8ab3c80562b8 37 lru 00000080:00200000:2.0:1787827204.788781:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000004d18b97a 37: 1< 0 - - > 2< 151552 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788783:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000069359959 00000080:00200000:2.0:1787827204.788785:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 38. 00000080:00200000:2.0:1787827204.788787:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 38 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788805:0:48442:0:(rw26.c:875:ll_write_end()) pos 155648, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788810:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000f304a0a5[3 00000000f3041499 2 1 00000000431eb59a 0 0 38] 00000080:00200000:2.0:1787827204.788813:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000079de4aca 17ffffc0001001 3:0 ffff8ab3c8056488 38 lru 00000080:00200000:2.0:1787827204.788818:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000cf428304 38: 1< 0 - - > 2< 155648 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788821:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000f304a0a5 00000080:00200000:2.0:1787827204.788822:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 39. 00000080:00200000:2.0:1787827204.788824:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 39 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788838:0:48442:0:(rw26.c:875:ll_write_end()) pos 159744, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788843:0:48442:0:(rw26.c:933:ll_write_end()) page@000000008169fbf5[3 00000000f3041499 2 1 00000000431eb59a 0 0 39] 00000080:00200000:2.0:1787827204.788846:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000fe9b0c7b 17ffffc0001001 3:0 ffff8ab3c80563a0 39 lru 00000080:00200000:2.0:1787827204.788851:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ddf0e51f 39: 1< 0 - - > 2< 159744 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788853:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000008169fbf5 00000080:00200000:2.0:1787827204.788855:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 40. 00000080:00200000:2.0:1787827204.788857:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 40 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788876:0:48442:0:(rw26.c:875:ll_write_end()) pos 163840, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788881:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000635fcfa2[3 00000000f3041499 2 1 00000000431eb59a 0 0 40] 00000080:00200000:2.0:1787827204.788884:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000faad7767 17ffffc0001021 2:0 ffff8ab3c80569f8 40 lru 00000080:00200000:2.0:1787827204.788906:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000002f81e51a 40: 1< 0 - - > 2< 163840 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788909:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000635fcfa2 00000080:00200000:2.0:1787827204.788910:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 41. 00000080:00200000:2.0:1787827204.788912:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 41 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788934:0:48442:0:(rw26.c:875:ll_write_end()) pos 167936, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788938:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000bca5cd50[3 00000000f3041499 2 1 00000000431eb59a 0 0 41] 00000080:00200000:2.0:1787827204.788941:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000edfa077e 17ffffc0001001 3:0 ffff8ab3c80560e8 41 lru 00000080:00200000:2.0:1787827204.788947:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000006a6b2876 41: 1< 0 - - > 2< 167936 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788949:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000bca5cd50 00000080:00200000:2.0:1787827204.788950:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 42. 00000080:00200000:2.0:1787827204.788952:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 42 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788967:0:48442:0:(rw26.c:875:ll_write_end()) pos 172032, len 4096, copied 4096 00000080:00200000:2.0:1787827204.788972:0:48442:0:(rw26.c:933:ll_write_end()) page@000000004e32ae7d[3 00000000f3041499 2 1 00000000431eb59a 0 0 42] 00000080:00200000:2.0:1787827204.788975:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000009d106d77 17ffffc0001001 3:0 ffff8ab3c8056740 42 lru 00000080:00200000:2.0:1787827204.788982:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000003d26389d 42: 1< 0 - - > 2< 172032 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.788984:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000004e32ae7d 00000080:00200000:2.0:1787827204.788986:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 43. 00000080:00200000:2.0:1787827204.788987:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 43 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.788999:0:48442:0:(rw26.c:875:ll_write_end()) pos 176128, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789002:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000a1f3a3c2[3 00000000f3041499 2 1 00000000431eb59a 0 0 43] 00000080:00200000:2.0:1787827204.789004:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000060952fcc 17ffffc0001001 3:0 ffff8ab3c8056570 43 lru 00000080:00200000:2.0:1787827204.789008:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ad31e66d 43: 1< 0 - - > 2< 176128 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789010:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000a1f3a3c2 00000080:00200000:2.0:1787827204.789011:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 44. 00000080:00200000:2.0:1787827204.789012:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 44 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789020:0:48442:0:(rw26.c:875:ll_write_end()) pos 180224, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789023:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000deb5da2a[3 00000000f3041499 2 1 00000000431eb59a 0 0 44] 00000080:00200000:2.0:1787827204.789024:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000001ff13aff 17ffffc0001001 3:0 ffff8ab3c80561d0 44 lru 00000080:00200000:2.0:1787827204.789029:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000ab1d6adc 44: 1< 0 - - > 2< 180224 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789030:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000deb5da2a 00000080:00200000:2.0:1787827204.789031:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 45. 00000080:00200000:2.0:1787827204.789032:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 45 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789041:0:48442:0:(rw26.c:875:ll_write_end()) pos 184320, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789043:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000042515e0[3 00000000f3041499 2 1 00000000431eb59a 0 0 45] 00000080:00200000:2.0:1787827204.789046:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000bdc8e0cd 17ffffc0001001 3:0 ffff8ab3c8056658 45 lru 00000080:00200000:2.0:1787827204.789050:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000979779ee 45: 1< 0 - - > 2< 184320 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789051:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000042515e0 00000080:00200000:2.0:1787827204.789052:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 46. 00000080:00200000:2.0:1787827204.789053:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 46 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789061:0:48442:0:(rw26.c:875:ll_write_end()) pos 188416, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789064:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000442c874d[3 00000000f3041499 2 1 00000000431eb59a 0 0 46] 00000080:00200000:2.0:1787827204.789066:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000b6eec0ed 17ffffc0001001 3:0 ffff8ab3c8056000 46 lru 00000080:00200000:2.0:1787827204.789069:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000b0264c2e 46: 1< 0 - - > 2< 188416 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789071:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000442c874d 00000080:00200000:2.0:1787827204.789072:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 47. 00000080:00200000:2.0:1787827204.789073:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 47 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789081:0:48442:0:(rw26.c:875:ll_write_end()) pos 192512, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789084:0:48442:0:(rw26.c:933:ll_write_end()) page@000000001c7b150f[3 00000000f3041499 2 1 00000000431eb59a 0 0 47] 00000080:00200000:2.0:1787827204.789086:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000f5aa95e6 17ffffc0001001 3:0 ffff8ab3c8056e80 47 lru 00000080:00200000:2.0:1787827204.789089:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000000b2a5cf 47: 1< 0 - - > 2< 192512 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789090:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000001c7b150f 00000080:00200000:2.0:1787827204.789091:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 48. 00000080:00200000:2.0:1787827204.789092:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 48 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789118:0:48442:0:(rw26.c:875:ll_write_end()) pos 196608, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789121:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000682fe597[3 00000000f3041499 2 1 00000000431eb59a 0 0 48] 00000080:00200000:2.0:1787827204.789124:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000508f1d04 17ffffc0001001 3:0 ffff8ab3c8056ae0 48 lru 00000080:00200000:2.0:1787827204.789128:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000047a92fae 48: 1< 0 - - > 2< 196608 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789129:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000682fe597 00000080:00200000:2.0:1787827204.789130:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 49. 00000080:00200000:2.0:1787827204.789132:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 49 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789140:0:48442:0:(rw26.c:875:ll_write_end()) pos 200704, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789143:0:48442:0:(rw26.c:933:ll_write_end()) page@000000002e232d1f[3 00000000f3041499 2 1 00000000431eb59a 0 0 49] 00000080:00200000:2.0:1787827204.789145:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000727abfee 17ffffc0001001 3:0 ffff8ab3c8056bc8 49 lru 00000080:00200000:2.0:1787827204.789149:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000004f1fa39f 49: 1< 0 - - > 2< 200704 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789151:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000002e232d1f 00000080:00200000:2.0:1787827204.789152:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 50. 00000080:00200000:2.0:1787827204.789153:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 50 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789162:0:48442:0:(rw26.c:875:ll_write_end()) pos 204800, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789165:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000f17ee898[3 00000000f3041499 2 1 00000000431eb59a 0 0 50] 00000080:00200000:2.0:1787827204.789167:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000001274490 17ffffc0001001 3:0 ffff8ab3c8056910 50 lru 00000080:00200000:2.0:1787827204.789170:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000071668803 50: 1< 0 - - > 2< 204800 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789172:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000f17ee898 00000080:00200000:2.0:1787827204.789173:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 51. 00000080:00200000:2.0:1787827204.789174:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 51 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789182:0:48442:0:(rw26.c:875:ll_write_end()) pos 208896, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789184:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000089870d09[3 00000000f3041499 2 1 00000000431eb59a 0 0 51] 00000080:00200000:2.0:1787827204.789186:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000b116628e 17ffffc0001001 3:0 ffff8ab3c8056828 51 lru 00000080:00200000:2.0:1787827204.789189:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000063df143f 51: 1< 0 - - > 2< 208896 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789191:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000089870d09 00000080:00200000:2.0:1787827204.789193:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 52. 00000080:00200000:2.0:1787827204.789194:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 52 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789208:0:48442:0:(rw26.c:875:ll_write_end()) pos 212992, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789213:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000bb44ef2e[3 00000000f3041499 2 1 00000000431eb59a 0 0 52] 00000080:00200000:2.0:1787827204.789217:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000f85eefdd 17ffffc0001001 3:0 ffff8ab3c8056cb0 52 lru 00000080:00200000:2.0:1787827204.789223:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000091ecf03b 52: 1< 0 - - > 2< 212992 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789226:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000bb44ef2e 00000080:00200000:2.0:1787827204.789227:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 53. 00000080:00200000:2.0:1787827204.789229:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 53 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789244:0:48442:0:(rw26.c:875:ll_write_end()) pos 217088, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789248:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000e485e7b3[3 00000000f3041499 2 1 00000000431eb59a 0 0 53] 00000080:00200000:2.0:1787827204.789250:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000002168b29f 17ffffc0001001 3:0 ffff8ab3c8056d98 53 lru 00000080:00200000:2.0:1787827204.789254:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000678d7d42 53: 1< 0 - - > 2< 217088 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789256:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000e485e7b3 00000080:00200000:2.0:1787827204.789257:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 54. 00000080:00200000:2.0:1787827204.789258:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 54 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789274:0:48442:0:(rw26.c:875:ll_write_end()) pos 221184, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789276:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000b0d0cd64[3 00000000f3041499 2 1 00000000431eb59a 0 0 54] 00000080:00200000:2.0:1787827204.789279:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000074c64d5e 17ffffc0001001 3:0 ffff8ab3d07d29f8 54 lru 00000080:00200000:2.0:1787827204.789283:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c35b9d44 54: 1< 0 - - > 2< 221184 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789284:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000b0d0cd64 00000080:00200000:2.0:1787827204.789285:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 55. 00000080:00200000:2.0:1787827204.789287:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 55 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789299:0:48442:0:(rw26.c:875:ll_write_end()) pos 225280, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789302:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000686cbd30[3 00000000f3041499 2 1 00000000431eb59a 0 0 55] 00000080:00200000:2.0:1787827204.789304:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000ec9572cc 17ffffc0001021 2:0 ffff8ab3d07d20e8 55 lru 00000080:00200000:2.0:1787827204.789307:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000d567c631 55: 1< 0 - - > 2< 225280 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789309:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000686cbd30 00000080:00200000:2.0:1787827204.789310:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 56. 00000080:00200000:2.0:1787827204.789311:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 56 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789320:0:48442:0:(rw26.c:875:ll_write_end()) pos 229376, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789322:0:48442:0:(rw26.c:933:ll_write_end()) page@000000004d72dcc5[3 00000000f3041499 2 1 00000000431eb59a 0 0 56] 00000080:00200000:2.0:1787827204.789324:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000b5e906e9 17ffffc0001001 3:0 ffff8ab3d07d2740 56 lru 00000080:00200000:2.0:1787827204.789327:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e7b08476 56: 1< 0 - - > 2< 229376 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789329:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000004d72dcc5 00000080:00200000:2.0:1787827204.789330:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 57. 00000080:00200000:2.0:1787827204.789331:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 57 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789340:0:48442:0:(rw26.c:875:ll_write_end()) pos 233472, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789343:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000020160f06[3 00000000f3041499 2 1 00000000431eb59a 0 0 57] 00000080:00200000:2.0:1787827204.789345:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000ce6b6295 17ffffc0001001 3:0 ffff8ab3d07d2570 57 lru 00000080:00200000:2.0:1787827204.789349:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000b8e88297 57: 1< 0 - - > 2< 233472 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789350:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000020160f06 00000080:00200000:2.0:1787827204.789351:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 58. 00000080:00200000:2.0:1787827204.789352:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 58 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789361:0:48442:0:(rw26.c:875:ll_write_end()) pos 237568, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789364:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000de67ec84[3 00000000f3041499 2 1 00000000431eb59a 0 0 58] 00000080:00200000:2.0:1787827204.789366:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000ec4ad1c0 17ffffc0001001 3:0 ffff8ab3d07d21d0 58 lru 00000080:00200000:2.0:1787827204.789369:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000019711cc 58: 1< 0 - - > 2< 237568 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789370:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000de67ec84 00000080:00200000:2.0:1787827204.789372:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 59. 00000080:00200000:2.0:1787827204.789373:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 59 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789381:0:48442:0:(rw26.c:875:ll_write_end()) pos 241664, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789384:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000a83dc058[3 00000000f3041499 2 1 00000000431eb59a 0 0 59] 00000080:00200000:2.0:1787827204.789385:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000205ec402 17ffffc0001001 3:0 ffff8ab3d07d2658 59 lru 00000080:00200000:2.0:1787827204.789389:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000080d66682 59: 1< 0 - - > 2< 241664 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789390:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000a83dc058 00000080:00200000:2.0:1787827204.789391:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 60. 00000080:00200000:2.0:1787827204.789392:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 60 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789401:0:48442:0:(rw26.c:875:ll_write_end()) pos 245760, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789403:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000068d6b745[3 00000000f3041499 2 1 00000000431eb59a 0 0 60] 00000080:00200000:2.0:1787827204.789406:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000004cf568b9 17ffffc0001001 3:0 ffff8ab3d07d2000 60 lru 00000080:00200000:2.0:1787827204.789410:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000003f8f98ac 60: 1< 0 - - > 2< 245760 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789411:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000068d6b745 00000080:00200000:2.0:1787827204.789412:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 61. 00000080:00200000:2.0:1787827204.789413:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 61 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789421:0:48442:0:(rw26.c:875:ll_write_end()) pos 249856, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789423:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000f710e185[3 00000000f3041499 2 1 00000000431eb59a 0 0 61] 00000080:00200000:2.0:1787827204.789425:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000721aebfc 17ffffc0001001 3:0 ffff8ab3d07d2e80 61 lru 00000080:00200000:2.0:1787827204.789429:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000003002b2c 61: 1< 0 - - > 2< 249856 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789430:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000f710e185 00000080:00200000:2.0:1787827204.789430:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 62. 00000080:00200000:2.0:1787827204.789432:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 62 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789441:0:48442:0:(rw26.c:875:ll_write_end()) pos 253952, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789444:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000deca2b40[3 00000000f3041499 2 1 00000000431eb59a 0 0 62] 00000080:00200000:2.0:1787827204.789446:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000000eb10e70 17ffffc0001001 3:0 ffff8ab3d07d2ae0 62 lru 00000080:00200000:2.0:1787827204.789449:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000d4e39316 62: 1< 0 - - > 2< 253952 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789450:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000deca2b40 00000080:00200000:2.0:1787827204.789451:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 63. 00000080:00200000:2.0:1787827204.789452:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 63 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789459:0:48442:0:(rw26.c:875:ll_write_end()) pos 258048, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789462:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000b6f9a0d1[3 00000000f3041499 2 1 00000000431eb59a 0 0 63] 00000080:00200000:2.0:1787827204.789465:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000af0a3496 17ffffc0001001 3:0 ffff8ab3d07d2bc8 63 lru 00000080:00200000:2.0:1787827204.789469:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000b304b746 63: 1< 0 - - > 2< 258048 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789470:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000b6f9a0d1 00000080:00200000:2.0:1787827204.789471:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 64. 00000080:00200000:2.0:1787827204.789472:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 64 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789483:0:48442:0:(rw26.c:875:ll_write_end()) pos 262144, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789486:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000022568211[3 00000000f3041499 2 1 00000000431eb59a 0 0 64] 00000080:00200000:2.0:1787827204.789488:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000e6f90349 17ffffc0001001 3:0 ffff8ab3d07d2910 64 lru 00000080:00200000:2.0:1787827204.789491:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000005cfb74c1 64: 1< 0 - - > 2< 262144 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789492:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000022568211 00000080:00200000:2.0:1787827204.789493:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 65. 00000080:00200000:2.0:1787827204.789494:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 65 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789503:0:48442:0:(rw26.c:875:ll_write_end()) pos 266240, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789506:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000014720ff9[3 00000000f3041499 2 1 00000000431eb59a 0 0 65] 00000080:00200000:2.0:1787827204.789508:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000ed9462bb 17ffffc0001001 3:0 ffff8ab3d07d2828 65 lru 00000080:00200000:2.0:1787827204.789512:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000048a8a8dd 65: 1< 0 - - > 2< 266240 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789513:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000014720ff9 00000080:00200000:2.0:1787827204.789514:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 66. 00000080:00200000:2.0:1787827204.789515:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 66 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789523:0:48442:0:(rw26.c:875:ll_write_end()) pos 270336, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789526:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000103038f7[3 00000000f3041499 2 1 00000000431eb59a 0 0 66] 00000080:00200000:2.0:1787827204.789527:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000006acd440c 17ffffc0001001 3:0 ffff8ab3d07d2cb0 66 lru 00000080:00200000:2.0:1787827204.789531:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000535c877e 66: 1< 0 - - > 2< 270336 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789532:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000103038f7 00000080:00200000:2.0:1787827204.789533:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 67. 00000080:00200000:2.0:1787827204.789533:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 67 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789542:0:48442:0:(rw26.c:875:ll_write_end()) pos 274432, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789544:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000024e3105c[3 00000000f3041499 2 1 00000000431eb59a 0 0 67] 00000080:00200000:2.0:1787827204.789547:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000d8559e8b 17ffffc0001001 3:0 ffff8ab3d07d2d98 67 lru 00000080:00200000:2.0:1787827204.789550:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000049d25dc8 67: 1< 0 - - > 2< 274432 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789552:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000024e3105c 00000080:00200000:2.0:1787827204.789553:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 68. 00000080:00200000:2.0:1787827204.789554:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 68 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789562:0:48442:0:(rw26.c:875:ll_write_end()) pos 278528, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789565:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000060f36441[3 00000000f3041499 2 1 00000000431eb59a 0 0 68] 00000080:00200000:2.0:1787827204.789567:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000fe495faf 17ffffc0001001 3:0 ffff8ab3d07d22b8 68 lru 00000080:00200000:2.0:1787827204.789571:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000079e2cc96 68: 1< 0 - - > 2< 278528 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789573:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000060f36441 00000080:00200000:2.0:1787827204.789574:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 69. 00000080:00200000:2.0:1787827204.789575:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 69 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789584:0:48442:0:(rw26.c:875:ll_write_end()) pos 282624, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789587:0:48442:0:(rw26.c:933:ll_write_end()) page@000000001083164d[3 00000000f3041499 2 1 00000000431eb59a 0 0 69] 00000080:00200000:2.0:1787827204.789588:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000001361884b 17ffffc0001001 3:0 ffff8ab3d07d2488 69 lru 00000080:00200000:2.0:1787827204.789592:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000004f2202e7 69: 1< 0 - - > 2< 282624 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789594:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000001083164d 00000080:00200000:2.0:1787827204.789595:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 70. 00000080:00200000:2.0:1787827204.789596:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 70 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789607:0:48442:0:(rw26.c:875:ll_write_end()) pos 286720, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789610:0:48442:0:(rw26.c:933:ll_write_end()) page@000000006837fefc[3 00000000f3041499 2 1 00000000431eb59a 0 0 70] 00000080:00200000:2.0:1787827204.789612:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000c68d8541 17ffffc0001021 2:0 ffff8ab3d07d23a0 70 lru 00000080:00200000:2.0:1787827204.789616:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000063957ee6 70: 1< 0 - - > 2< 286720 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789618:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000006837fefc 00000080:00200000:2.0:1787827204.789618:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 71. 00000080:00200000:2.0:1787827204.789619:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 71 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789635:0:48442:0:(rw26.c:875:ll_write_end()) pos 290816, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789637:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000e01caa57[3 00000000f3041499 2 1 00000000431eb59a 0 0 71] 00000080:00200000:2.0:1787827204.789639:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000b0b72190 17ffffc0001001 3:0 ffff8ab3c6a9c488 71 lru 00000080:00200000:2.0:1787827204.789643:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000485a2068 71: 1< 0 - - > 2< 290816 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789644:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000e01caa57 00000080:00200000:2.0:1787827204.789645:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 72. 00000080:00200000:2.0:1787827204.789646:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 72 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789655:0:48442:0:(rw26.c:875:ll_write_end()) pos 294912, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789658:0:48442:0:(rw26.c:933:ll_write_end()) page@000000007a98871d[3 00000000f3041499 2 1 00000000431eb59a 0 0 72] 00000080:00200000:2.0:1787827204.789660:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000009f2308a1 17ffffc0001001 3:0 ffff8ab3c6a9c3a0 72 lru 00000080:00200000:2.0:1787827204.789664:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000cb667953 72: 1< 0 - - > 2< 294912 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789666:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000007a98871d 00000080:00200000:2.0:1787827204.789667:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 73. 00000080:00200000:2.0:1787827204.789668:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 73 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789676:0:48442:0:(rw26.c:875:ll_write_end()) pos 299008, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789679:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000a11212e7[3 00000000f3041499 2 1 00000000431eb59a 0 0 73] 00000080:00200000:2.0:1787827204.789681:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000a216d482 17ffffc0001001 3:0 ffff8ab3c6a9c9f8 73 lru 00000080:00200000:2.0:1787827204.789684:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@0000000039815c93 73: 1< 0 - - > 2< 299008 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789686:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000a11212e7 00000080:00200000:2.0:1787827204.789687:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 74. 00000080:00200000:2.0:1787827204.789688:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 74 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789697:0:48442:0:(rw26.c:875:ll_write_end()) pos 303104, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789699:0:48442:0:(rw26.c:933:ll_write_end()) page@000000000ff4c277[3 00000000f3041499 2 1 00000000431eb59a 0 0 74] 00000080:00200000:2.0:1787827204.789701:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000b9aa68cb 17ffffc0001001 3:0 ffff8ab3c6a9c0e8 74 lru 00000080:00200000:2.0:1787827204.789705:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000007a9911af 74: 1< 0 - - > 2< 303104 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789706:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000000ff4c277 00000080:00200000:2.0:1787827204.789706:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 75. 00000080:00200000:2.0:1787827204.789707:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 75 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789716:0:48442:0:(rw26.c:875:ll_write_end()) pos 307200, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789718:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000d895e4ce[3 00000000f3041499 2 1 00000000431eb59a 0 0 75] 00000080:00200000:2.0:1787827204.789720:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @0000000038d0159f 17ffffc0001001 3:0 ffff8ab3c6a9c740 75 lru 00000080:00200000:2.0:1787827204.789723:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000e2481c5b 75: 1< 0 - - > 2< 307200 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789724:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000d895e4ce 00000080:00200000:2.0:1787827204.789725:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 76. 00000080:00200000:2.0:1787827204.789726:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 76 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.789736:0:48442:0:(rw26.c:875:ll_write_end()) pos 311296, len 4096, copied 4096 00000080:00200000:2.0:1787827204.789738:0:48442:0:(rw26.c:933:ll_write_end()) page@000000007a53a64b[3 00000000f3041499 2 1 00000000431eb59a 0 0 76] 00000080:00200000:2.0:1787827204.789740:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000ead4da75 17ffffc0001001 3:0 ffff8ab3c6a9c570 76 lru 00000080:00200000:2.0:1787827204.789744:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000fbaa2929 76: 1< 0 - - > 2< 311296 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789745:0:48442:0:(rw26.c:933:ll_write_end()) end page@000000007a53a64b 00000080:00200000:2.0:1787827204.789747:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 77. 00000080:00200000:2.0:1787827204.789748:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 77 of 0 to 2156 bytes 00000080:00200000:2.0:1787827204.789761:0:48442:0:(rw26.c:875:ll_write_end()) pos 315392, len 2156, copied 2156 00000080:00200000:2.0:1787827204.789764:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000e0f7e611[3 00000000f3041499 2 1 00000000431eb59a 0 0 77] 00000080:00200000:2.0:1787827204.789766:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @00000000e3c288ae 17ffffc0001001 3:0 ffff8ab3c6a9c1d0 77 lru 00000080:00200000:2.0:1787827204.789769:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@00000000c07e7b83 77: 1< 0 - - > 2< 315392 0 0 0x0 0x0 | 00000000f5856a27 000000005f25686e > 3< 0 0 > 4< 0 0 8 75923456 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.789771:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000e0f7e611 00000080:00200000:2.0:1787827204.789772:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 78. 00000080:00200000:2.0:1787827204.789775:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 78, from 0, to 2156 prio 0 00000008:00010000:2.0:1787827204.789793:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789808:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789815:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789820:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789825:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789829:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789834:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789845:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789850:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789855:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789860:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789865:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789870:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789874:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789879:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.789911:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f13f1b38, count 15, dirtied 15 00000008:00010000:2.0:1787827204.789918:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789926:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789932:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789938:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789943:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789948:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789953:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789958:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789962:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789972:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789977:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789982:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789986:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789991:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.789996:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.790003:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f13f1b38, count 15, dirtied 15 00000008:00010000:2.0:1787827204.790006:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790011:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790019:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790024:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790029:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790034:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790038:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790043:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790048:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790053:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790058:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790063:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790068:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790073:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790077:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.790085:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f13f1b38, count 15, dirtied 15 00000008:00010000:2.0:1787827204.790088:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790105:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790112:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790118:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790127:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790132:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790137:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790142:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790147:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790152:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790156:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790161:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790166:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790172:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790178:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.790185:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f13f1b38, count 15, dirtied 15 00000008:00010000:2.0:1787827204.790188:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790195:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790206:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790216:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790226:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790236:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790252:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790261:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790271:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790281:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790288:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790293:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790298:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790303:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790308:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.790317:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f13f1b38, count 15, dirtied 15 00000008:00010000:2.0:1787827204.790320:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790327:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.790332:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.790338:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f13f1b38, count 3, dirtied 3 00000080:00200000:2.0:1787827204.790341:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 78 pages 317548 bytes, tot: 317548 00000080:00200000:2.0:1787827204.790343:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) test-framework.sh: write bytes 0, result: 317548 00000020:00010000:2.0:1787827204.790354:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3c6a25d58 (ffff8ab3c6ae1908/0) at cl_lock_release():245 00000020:00010000:2.0:1787827204.790357:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3c6a25d58 (ffff8ab3c6ae1908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.790358:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d068e4b0 (ffff8ab3c6ae1d88/1) at cl_lock_cancel():142 00010000:00010000:2.0:1787827204.790362:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.790369:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1787827204.790379:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3c6a25d58 (ffff8ab3c6ae1908/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827204.790381:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d068e4b0 (ffff8ab3c6ae1908/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827204.790387:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 317548 bytes 00000080:00200000:2.0:1787827204.790392:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 77] 00000080:00200000:2.0:1787827204.790398:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4d:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827204.790402:0:48442:0:(file.c:2240:ll_file_io_generic()) test-framework.sh: 2 io complete with rc: 0, result: 317548, restart: 0 00000080:00200000:2.0:1787827204.790408:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x4d:0x0] updating i_size 317548 i_blocks 0 00000080:00200000:2.0:1787827204.790413:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 317548 00000080:00200008:2.0:1787827204.790438:0:48442:0:(file.c:479:ll_file_release()) START file test-framework.sh:[0x240001b71:0x4d:0x0](ffff8ab3f13f1988), flags 100001 00000002:00100000:2.0:1787827204.790475:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3d22ff0c0 x1874671280568832/t17179869526(17179869526) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 592/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787827204.790484:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787827204.790488:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c69f43c0 x1874671280569472/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787827204.790500:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f43c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280569472:192.168.203.124@tcp:35:cp.0 00000100:00100000:2.0:1787827204.790537:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913840 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.791424:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f43c0 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280569472:192.168.203.124@tcp:35:cp.0 00000002:00100000:2.0:1787827204.791446:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3d22ff0c0 x1874671280568832/t17179869526(17179869526) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 592/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787827204.791497:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.791510:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.791523:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.791532:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.791537:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.791542:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.791544:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.791550:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:2.0:1787827204.791552:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.791558:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.791566:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.791571:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.791575:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.791578:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.791580:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.791582:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=test-framework.sh, flags=8 00000080:00200008:2.0:1787827204.791587:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file test-framework.sh:[0x240001b71:0x4d:0x0](ffff8ab3f13f1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827204.791591:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x4d:0x0](ffff8ab3f13f1988),name=test-framework.sh 00000002:00010000:2.0:1787827204.791598:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x4d:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.791603:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854600/0x268e1a602cc2574 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da80b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827204.791609:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854600/0x268e1a602cc2574 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da80b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827204.791614:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.791616:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x4d:0x0](ffff8ab3f13f1988) 00000080:00010000:2.0:1787827204.791618:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x4d:0x0](ffff8ab3f13f1988) for lock 0x268e1a602cc2574 00000080:00010000:2.0:1787827204.791620:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2574 from it ffffb86e87e83cb8 00010000:00010000:2.0:1787827204.791622:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854600/0x268e1a602cc2574 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da80b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827204.791631:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854600/0x268e1a602cc2574 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da80b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827204.791642:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827204.791647:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x4d:0x0] 00000008:00010000:2.0:1787827204.791655:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827204.791657:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d068e4b0, osc lock ffff8ab3c6ab1728, flags 0x400001002 00010000:00010000:2.0:1787827204.791664:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.791672:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1787827204.791679:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827204.791682:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.791688:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1787827204.791693:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827204.791697:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x4d:0x0] updating i_size 317548 i_blocks 0 00000020:00010000:2.0:1787827204.791700:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3c4942018 (ffff8ab3c6ae1908/0) at cl_lock_release():245 00000020:00010000:2.0:1787827204.791702:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3c4942018 (ffff8ab3c6ae1908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.791704:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d068e4b0 (ffff8ab3c6ae1d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.791706:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3c4942018 (ffff8ab3c6ae1908/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827204.791708:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d068e4b0 (ffff8ab3c6ae1908/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827204.791714:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4d:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827204.791746:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.791752:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.791756:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.791763:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.791767:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.791770:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.791772:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.791775:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:2.0:1787827204.791777:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.791782:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.791787:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.791789:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:2.0:1787827204.791792:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.791794:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1787827204.791795:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.791804:0:48442:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=yaml.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=9 00000080:00200000:2.0:1787827204.791808:0:48442:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=yaml.sh, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=getattr 00000002:00010000:2.0:1787827204.791814:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: yaml.sh,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827204.791838:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68800/0x268e1a602cc2589 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.791843:0:48442:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68800/0x268e1a602cc2589 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.791850:0:48442:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68800/0x268e1a602cc2589 lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48442 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827204.791859:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f6d00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280569600:192.168.203.124@tcp:101:cp.0 00000100:00100000:2.0:1787827204.791921:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913840 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.796411:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280570112:192.168.203.124@tcp:101:cp.0 00010000:00010000:0.0:1787827204.797207:0:10956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.797220:0:10956:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da874 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.797230:0:10956:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da874 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.797238:0:10956:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da874 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1787827204.797245:0:10956:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da874 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1787827204.797252:0:10956:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1787827204.797262:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da874 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.797268:0:10956:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da874 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.797273:0:10956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da874 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827204.797279:0:10956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da874 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787827204.797296:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280570112:192.168.203.124@tcp:101:cp.0 00000100:00100000:3.0:1787827204.798121:0:10948:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00000100:00100000:2.0:1787827204.798162:0:47646:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874671294805760 00000100:00100000:2.0:1787827204.798168:0:47646:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874671294805760 00000100:00100000:2.0:1787827204.798205:0:47646:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.124@tcp, seq: 308 00000100:00100000:2.0:1787827204.798213:0:47646:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8ab3e0ba61c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:82330:x1874671294805760:12345-192.168.203.124@tcp:104: 00010000:00010000:2.0:1787827204.798222:0:47646:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da874 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.798261:0:47646:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1787827204.798267:0:47646:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8ab3e0ba61c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:82330:x1874671294805760:12345-192.168.203.124@tcp:104: Request processed in 55us (148us total) trans 0 rc 0/0 00000100:00100000:2.0:1787827204.798273:0:47646:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.124@tcp, seq: 308 00010000:00010000:0.0:1787827204.798285:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1787827204.798302:0:47650:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x772721ad1f0da874 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.798318:0:47650:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8ab3d0855a00 already unused, calling callback (ffffffffc10b6410) 00010000:00010000:0.0:1787827204.798324:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x772721ad1f0da874 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827204.798340:0:47650:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827204.798348:0:47650:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787827204.798377:0:47650:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da874 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.798386:0:47650:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787827204.798409:0:47650:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da874 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827204.798420:0:47650:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000057c6df28/0x268e1a602cc259e lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0xa8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x772721ad1f0da874 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1787827204.798435:0:10956:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00010000:00010000:0.0:1787827204.798437:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827204.798441:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827204.798443:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827204.798449:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aada40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280570368:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1787827204.799284:0:10949:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.124@tcp (source: 12345-192.168.203.124@tcp) 00010000:00010000:2.0:1787827204.799470:0:48442:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d707e436/0x268e1a602cc25a5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da87b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.799486:0:48442:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d707e436/0x268e1a602cc25a5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x772721ad1f0da87b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.799497:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d707e436/0x268e1a602cc25a5 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x772721ad1f0da87b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.799505:0:48442:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d707e436/0x268e1a602cc25a5 lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10020000020000 nid: local remote: 0x772721ad1f0da87b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.799513:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d707e436/0x268e1a602cc25a5 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da87b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.799518:0:48442:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d707e436/0x268e1a602cc25a5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da87b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.799533:0:48442:0:(vvp_io.c:1339:vvp_io_write_start()) yaml.sh: write [0, 4512) append 0 append lockpos 0 00000080:00200000:2.0:1787827204.799545:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.799580:0:48442:0:(rw26.c:875:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1787827204.799599:0:48442:0:(rw26.c:933:ll_write_end()) page@0000000023b9caeb[3 00000000d254d1f5 2 1 000000007cb5813e 0 0 0] 00000080:00200000:2.0:1787827204.799602:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000008d9e7650 17ffffc0001001 3:0 ffff8ab3c6a9c658 0 lru 00000080:00200000:2.0:1787827204.799608:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000000dd82728 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000006a588f83 0000000016fc7052 > 3< 0 0 > 4< 0 0 8 2424832 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.799609:0:48442:0:(rw26.c:933:ll_write_end()) end page@0000000023b9caeb 00000080:00200000:2.0:1787827204.799610:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1787827204.799613:0:48442:0:(rw26.c:681:ll_write_begin()) Writing 1 of 0 to 416 bytes 00000080:00200000:2.0:1787827204.799632:0:48442:0:(rw26.c:875:ll_write_end()) pos 4096, len 416, copied 416 00000080:00200000:2.0:1787827204.799636:0:48442:0:(rw26.c:933:ll_write_end()) page@00000000a03c98fc[3 00000000d254d1f5 2 1 000000007cb5813e 0 0 1] 00000080:00200000:2.0:1787827204.799638:0:48442:0:(rw26.c:933:ll_write_end()) vmpage @000000008a63d015 17ffffc0001001 3:0 ffff8ab3c6a9c000 1 lru 00000080:00200000:2.0:1787827204.799643:0:48442:0:(rw26.c:933:ll_write_end()) osc-page@000000006cf2377a 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000006a588f83 0000000016fc7052 > 3< 0 0 > 4< 0 0 8 2424832 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.799644:0:48442:0:(rw26.c:933:ll_write_end()) end page@00000000a03c98fc 00000080:00200000:2.0:1787827204.799645:0:48442:0:(rw26.c:933:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1787827204.799648:0:48442:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 2, from 0, to 416 prio 0 00000008:00010000:2.0:1787827204.799663:0:48442:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000a04130da ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d707e436/0x268e1a602cc25a5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da87b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.799675:0:48442:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 00000000a04130da ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d707e436/0x268e1a602cc25a5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da87b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.799685:0:48442:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f13f2bb8, count 2, dirtied 2 00000080:00200000:2.0:1787827204.799688:0:48442:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 2 pages 4512 bytes, tot: 4512 00000080:00200000:2.0:1787827204.799690:0:48442:0:(vvp_io.c:1421:vvp_io_write_start()) yaml.sh: write bytes 0, result: 4512 00000020:00010000:2.0:1787827204.799700:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3c6a258a8 (ffff8ab3c6ae12d8/0) at cl_lock_release():245 00000020:00010000:2.0:1787827204.799702:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3c6a258a8 (ffff8ab3c6ae12d8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.799704:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3c3760530 (ffff8ab3c6ae1ab8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1787827204.799708:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d707e436/0x268e1a602cc25a5 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da87b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.799714:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d707e436/0x268e1a602cc25a5 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da87b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1787827204.799723:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3c6a258a8 (ffff8ab3c6ae12d8/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827204.799726:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3c3760530 (ffff8ab3c6ae12d8/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827204.799732:0:48442:0:(vvp_io.c:533:vvp_io_advance()) advancing 4512 bytes 00000080:00200000:2.0:1787827204.799737:0:48442:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 1] 00000080:00200000:2.0:1787827204.799742:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4e:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827204.799747:0:48442:0:(file.c:2240:ll_file_io_generic()) yaml.sh: 2 io complete with rc: 0, result: 4512, restart: 0 00000080:00200000:2.0:1787827204.799752:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x4e:0x0] updating i_size 4512 i_blocks 0 00000080:00200000:2.0:1787827204.799758:0:48442:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 4512 00000080:00200008:2.0:1787827204.799777:0:48442:0:(file.c:479:ll_file_release()) START file yaml.sh:[0x240001b71:0x4e:0x0](ffff8ab3f13f2a08), flags 100001 00000002:00100000:2.0:1787827204.799810:0:48442:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3d22fcf00 x1874671280569984/t17179869528(17179869528) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 576/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787827204.799818:0:48442:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787827204.799822:0:48442:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c69f6d00 x1874671280570624/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787827204.799833:0:48442:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f6d00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280570624:192.168.203.124@tcp:35:cp.0 00000100:00100000:2.0:1787827204.799864:0:48442:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913780 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.800029:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aada40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280570368:192.168.203.124@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787827204.800746:0:48442:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f6d00 pname:cluuid:pid:xid:nid:opc:job cp:27aedc06-3022-4c11-aafb-6bd27e69b836:48442:1874671280570624:192.168.203.124@tcp:35:cp.0 00000002:00100000:2.0:1787827204.800761:0:48442:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3d22fcf00 x1874671280569984/t17179869528(17179869528) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 576/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787827204.800795:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.800805:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.800813:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.800821:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.800826:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.800830:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.800832:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.800836:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e83b58 00010000:00010000:2.0:1787827204.800839:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.800850:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.800857:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.800862:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.800865:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.800867:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.800869:0:48442:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.800871:0:48442:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=yaml.sh, flags=8 00000080:00200008:2.0:1787827204.800875:0:48442:0:(file.c:6313:ll_getattr_dentry()) START file yaml.sh:[0x240001b71:0x4e:0x0](ffff8ab3f13f2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827204.800878:0:48442:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x240001b71:0x4e:0x0](ffff8ab3f13f2a08),name=yaml.sh 00000002:00010000:2.0:1787827204.800911:0:48442:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x4e:0x0]) in obj [0x240001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.800920:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854200/0x268e1a602cc2597 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da866 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827204.800930:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854200/0x268e1a602cc2597 lrc: 2/1,0 mode: PR/PR res: [0x240001b71:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da866 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827204.800939:0:48442:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240001b71:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.800944:0:48442:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x4e:0x0](ffff8ab3f13f2a08) 00000080:00010000:2.0:1787827204.800947:0:48442:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x4e:0x0](ffff8ab3f13f2a08) for lock 0x268e1a602cc2597 00000080:00010000:2.0:1787827204.800952:0:48442:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc2597 from it ffffb86e87e83cb8 00010000:00010000:2.0:1787827204.800955:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854200/0x268e1a602cc2597 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da866 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827204.800961:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854200/0x268e1a602cc2597 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da866 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827204.800972:0:48442:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827204.800977:0:48442:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240001b71:0x4e:0x0] 00000008:00010000:2.0:1787827204.800985:0:48442:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827204.800987:0:48442:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d068e4b0, osc lock ffff8ab3c6ab1728, flags 0x400001002 00010000:00010000:2.0:1787827204.800995:0:48442:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d707e436/0x268e1a602cc25a5 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0xa8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da87b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.801002:0:48442:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d707e436/0x268e1a602cc25a5 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da87b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1787827204.801008:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827204.801011:0:48442:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d707e436/0x268e1a602cc25a5 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x772721ad1f0da87b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.801016:0:48442:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d707e436/0x268e1a602cc25a5 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da87b expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1787827204.801022:0:48442:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827204.801026:0:48442:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x4e:0x0] updating i_size 4512 i_blocks 0 00000020:00010000:2.0:1787827204.801028:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3c4942780 (ffff8ab3c6ae12d8/0) at cl_lock_release():245 00000020:00010000:2.0:1787827204.801030:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3c4942780 (ffff8ab3c6ae12d8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.801032:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d068e4b0 (ffff8ab3c6ae1ab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.801034:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3c4942780 (ffff8ab3c6ae12d8/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827204.801036:0:48442:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d068e4b0 (ffff8ab3c6ae12d8/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827204.801042:0:48442:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827204.805702:0:48443:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.805717:0:48443:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.805729:0:48443:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.805744:0:48443:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.805753:0:48443:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.805760:0:48443:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.805763:0:48443:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.805770:0:48443:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e63e18 00010000:00010000:2.0:1787827204.805775:0:48443:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.805785:0:48443:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.805796:0:48443:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 101 00000080:00200000:2.0:1787827204.805803:0:48443:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.805809:0:48443:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.805814:0:48443:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.805820:0:48443:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.805825:0:48443:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.805828:0:48443:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.805829:0:48443:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.805832:0:48443:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e63be0 00010000:00010000:2.0:1787827204.805835:0:48443:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.805840:0:48443:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.805845:0:48443:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.805849:0:48443:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=1536 00000080:00200000:2.0:1787827204.805854:0:48443:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.805858:0:48443:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.805861:0:48443:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.805866:0:48443:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.805871:0:48443:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.805873:0:48443:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.805875:0:48443:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.805877:0:48443:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e63c18 00010000:00010000:2.0:1787827204.805878:0:48443:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.805883:0:48443:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.805914:0:48443:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.805920:0:48443:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=3 00000080:00200000:2.0:1787827204.805927:0:48443:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 101 00000080:00200000:2.0:1787827204.805936:0:48443:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.805940:0:48443:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=1536 00000080:00200000:2.0:1787827204.805945:0:48443:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.805947:0:48443:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=3 00000080:00200000:2.0:1787827204.805949:0:48443:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 101 00000080:00200000:2.0:1787827204.805953:0:48443:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.805959:0:48443:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d_f6.sanity-scrub, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.805961:0:48443:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.805964:0:48443:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.805968:0:48443:0:(namei.c:2239:do_mkdir()) VFS Op:name=d_f6.sanity-scrub, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) 00000020:00100000:2.0:1787827204.806012:0:48443:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.806021:0:48443:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c69f4000 x1874671280570880/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.806036:0:48443:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c69f4000 pname:cluuid:pid:xid:nid:opc:job mkdir:27aedc06-3022-4c11-aafb-6bd27e69b836:48443:1874671280570880:192.168.203.124@tcp:36:mkdir.0 00000100:00100000:2.0:1787827204.806086:0:48443:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913780 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.807952:0:48443:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c69f4000 pname:cluuid:pid:xid:nid:opc:job mkdir:27aedc06-3022-4c11-aafb-6bd27e69b836:48443:1874671280570880:192.168.203.124@tcp:36:mkdir.0 00000080:00200000:2.0:1787827204.807980:0:48443:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x4f:0x0](ffff8ab3f11fa1c8) 00000080:00200000:2.0:1787827204.807985:0:48443:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240001b71:0x4f:0x0], updating i_size 4096 00000080:00200000:2.0:1787827204.811228:0:48444:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.811247:0:48444:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.811264:0:48444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.811277:0:48444:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.811291:0:48444:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.811298:0:48444:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.811301:0:48444:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.811308:0:48444:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e23bf8 00010000:00010000:2.0:1787827204.811313:0:48444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.811321:0:48444:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.811331:0:48444:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.811336:0:48444:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:2.0:1787827204.811359:0:48444:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.811362:0:48444:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1787827204.811364:0:48444:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.811371:0:48444:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d_f6.sanity-scrub, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=25 00000080:00200000:2.0:1787827204.811375:0:48444:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d_f6.sanity-scrub, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), intent=0 00000002:00010000:2.0:1787827204.811385:0:48444:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d_f6.sanity-scrub,[0x0:0x0:0x0]) in obj [0x240001b71:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1787827204.811418:0:48444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.811425:0:48444:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.811433:0:48444:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48444 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827204.811447:0:48444:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0645e00 pname:cluuid:pid:xid:nid:opc:job touch:27aedc06-3022-4c11-aafb-6bd27e69b836:48444:1874671280571008:192.168.203.124@tcp:101:touch.0 00000100:00100000:2.0:1787827204.811507:0:48444:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913780 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.812693:0:48444:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0645e00 pname:cluuid:pid:xid:nid:opc:job touch:27aedc06-3022-4c11-aafb-6bd27e69b836:48444:1874671280571008:192.168.203.124@tcp:101:touch.0 00010000:00010000:2.0:1787827204.812709:0:48444:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 4/1,0 mode: --/CR res: [0x240001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8ac expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.812722:0:48444:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 4/1,0 mode: --/PR res: [0x240001b71:0x4f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8ac expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.812731:0:48444:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 4/1,0 mode: --/PR res: [0x240001b71:0x4f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8ac expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.812744:0:48444:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x4f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8ac expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.812756:0:48444:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x4f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8ac expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.812769:0:48444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 5/2,0 mode: PR/PR res: [0x240001b71:0x4f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8ac expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.812779:0:48444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 5/2,0 mode: PR/PR res: [0x240001b71:0x4f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8ac expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.812784:0:48444:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x4f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8ac expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827204.812793:0:48444:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8ab3d0645e00 x1874671280571008/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 592/584 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827204.812804:0:48444:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x4f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8ac expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.812811:0:48444:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/79 (0 0) 00000080:00200000:2.0:1787827204.812814:0:48444:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x240001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827204.812826:0:48444:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240001b71:0x4f:0x0], updating i_size 4096 00000080:00200000:2.0:1787827204.812829:0:48444:0:(namei.c:147:ll_iget()) got inode: [0x240001b71:0x4f:0x0](ffff8ab3f11fa1c8): rc = 0 00000080:00010000:2.0:1787827204.812833:0:48444:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x4f:0x0](ffff8ab3f11fa1c8) for lock 0x268e1a602cc25ac 00000080:00010000:2.0:1787827204.812861:0:48444:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x4f:0x0](ffff8ab3f11fa1c8) 00000080:00010000:2.0:1787827204.812863:0:48444:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc25ac from it ffffb86e87e23ac8 00010000:00010000:2.0:1787827204.812866:0:48444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x4f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8ac expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.812872:0:48444:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x4f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8ac expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827204.812909:0:48444:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x4f:0x0](ffff8ab3f11fa1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.812917:0:48444:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x4f:0x0](ffff8ab3f11fa1c8), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.812921:0:48444:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f1, dir=[0x240001b71:0x4f:0x0](ffff8ab3f11fa1c8), file ffff8ab3d03ad680, open_flags 8941, mode 81b6 opened 0 00000080:00200000:2.0:1787827204.812928:0:48444:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f1, dir=[0x240001b71:0x4f:0x0](ffff8ab3f11fa1c8), intent=open|creat 00000002:00010000:2.0:1787827204.812939:0:48444:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f1,[0x240001b71:0x50:0x0]) in obj [0x240001b71:0x4f:0x0], intent: open|creat flags 0104502 00010000:00010000:2.0:1787827204.812961:0:48444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x4f:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.812966:0:48444:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x4f:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48444 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787827204.812977:0:48444:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787827204.812982:0:48444:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0645e00 x1874671280571264/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787827204.812989:0:48444:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x4f:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48444 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827204.812997:0:48444:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0645e00 pname:cluuid:pid:xid:nid:opc:job touch:27aedc06-3022-4c11-aafb-6bd27e69b836:48444:1874671280571264:192.168.203.124@tcp:101:touch.0 00000100:00100000:2.0:1787827204.813121:0:48444:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913780 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.814367:0:48444:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0645e00 pname:cluuid:pid:xid:nid:opc:job touch:27aedc06-3022-4c11-aafb-6bd27e69b836:48444:1874671280571264:192.168.203.124@tcp:101:touch.0 00010000:00010000:2.0:1787827204.814382:0:48444:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x4f:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.814397:0:48444:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x50:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.814406:0:48444:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.814417:0:48444:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.814427:0:48444:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.814439:0:48444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.814456:0:48444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.814465:0:48444:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827204.814478:0:48444:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3d0645e00 x1874671280571264/t17179869531(17179869531) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 648/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787827204.814492:0:48444:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0645e00 x1874671280571264/t17179869531(17179869531) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 648/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827204.814507:0:48444:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827204.814519:0:48444:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.814528:0:48444:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/80 (0 0) 00000080:00200000:2.0:1787827204.814533:0:48444:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x4f:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827204.814541:0:48444:0:(namei.c:1628:ll_create_it()) VFS Op:name=f1, dir=[0x240001b71:0x4f:0x0](ffff8ab3f11fa1c8), intent=open|creat 00000080:00200000:2.0:1787827204.814561:0:48444:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x50:0x0](ffff8ab3f11fc2c8) 00000080:00200000:2.0:1787827204.814646:0:48444:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x50:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1787827204.814652:0:48444:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x50:0x0](ffff8ab3f11fc2c8) 00000080:00010000:2.0:1787827204.814657:0:48444:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x50:0x0](ffff8ab3f11fc2c8) for lock 0x268e1a602cc25b3 00000080:00010000:2.0:1787827204.814665:0:48444:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x50:0x0](ffff8ab3f11fc2c8) 00000080:00200008:2.0:1787827204.814674:0:48444:0:(file.c:1090:ll_file_open()) START file f1:[0x240001b71:0x50:0x0](ffff8ab3f11fc2c8), flags 104501 00000002:00100000:2.0:1787827204.814694:0:48444:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0645e00 x1874671280571264/t17179869531(17179869531) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 576/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787827204.814732:0:48444:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc25b3 from it ffff8ab3d0686120 00010000:00010000:2.0:1787827204.814738:0:48444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.814748:0:48444:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827204.814767:0:48444:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x50:0x0](ffff8ab3f11fc2c8), inode mode 81a4 mask 40 00000080:00200008:2.0:1787827204.814793:0:48444:0:(llite_lib.c:2427:ll_setattr_raw()) START file f1:[0x240001b71:0x50:0x0](ffff8ab3f11fc2c8) current size 0, valid attrs 20070, mode 8ab3, uid -2015216048, gid -18322, new size 18446744072032287255, atime 1787827204.000000000, mtime 1787827204.000000000, ctime 1787827204.000000000 00010000:00010000:2.0:1787827204.814813:0:48444:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827204.814830:0:48444:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x50:0x0]: losing layout lock 00010000:00010000:2.0:1787827204.814862:0:48444:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.814871:0:48444:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787827204.814875:0:48444:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25b3 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da8c8 expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1787827204.815072:0:48444:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.815081:0:48444:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a24f00 x1874671280571520/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.815115:0:48444:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a24f00 pname:cluuid:pid:xid:nid:opc:job touch:27aedc06-3022-4c11-aafb-6bd27e69b836:48444:1874671280571520:192.168.203.124@tcp:36:touch.0 00000100:00100000:2.0:1787827204.815167:0:48444:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913000 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.817179:0:48444:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a24f00 pname:cluuid:pid:xid:nid:opc:job touch:27aedc06-3022-4c11-aafb-6bd27e69b836:48444:1874671280571520:192.168.203.124@tcp:36:touch.0 00000080:00200000:2.0:1787827204.817224:0:48444:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x50:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1787827204.817250:0:48444:0:(file.c:7076:ll_layout_intent()) ### lustre: requeue layout lock for file [0x240001b71:0x50:0x0](ffff8ab3f11fc2c8) 00000002:00010000:2.0:1787827204.817259:0:48444:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240001b71:0x50:0x0]) in obj [0x240001b71:0x50:0x0], intent: layout flags 0 00010000:00010000:2.0:1787827204.817316:0:48444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 48444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.817324:0:48444:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 48444 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.817332:0:48444:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 3/1,0 mode: --/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 48444 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827204.817346:0:48444:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a26580 pname:cluuid:pid:xid:nid:opc:job touch:27aedc06-3022-4c11-aafb-6bd27e69b836:48444:1874671280571648:192.168.203.124@tcp:101:touch.0 00000100:00100000:2.0:1787827204.817394:0:48444:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913000 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.818176:0:48444:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a26580 pname:cluuid:pid:xid:nid:opc:job touch:27aedc06-3022-4c11-aafb-6bd27e69b836:48444:1874671280571648:192.168.203.124@tcp:101:touch.0 00010000:00010000:2.0:1787827204.818192:0:48444:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 4/1,0 mode: CR/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8dd expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.818201:0:48444:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 4/1,0 mode: CR/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8dd expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827204.818213:0:48444:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff8ab3d0a26580 x1874671280571648/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 376/400 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827204.818221:0:48444:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 3/1,0 mode: CR/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8dd expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827204.818234:0:48444:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 3/1,0 mode: CR/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8dd expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.818242:0:48444:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 16 res 9663683441/80 (0 0) 00000080:00200000:2.0:1787827204.818246:0:48444:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x240001b71:0x50:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1787827204.818261:0:48444:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x50:0x0](ffff8ab3f11fc2c8) for lock 0x268e1a602cc25ba 00000080:00010000:2.0:1787827204.818267:0:48444:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc25ba from it ffffb86e87e23a78 00010000:00010000:2.0:1787827204.818270:0:48444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 3/1,0 mode: CR/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8dd expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.818277:0:48444:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 2/0,0 mode: CR/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8dd expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.818290:0:48444:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 2/1,0 mode: CR/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8dd expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.818296:0:48444:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 2/1,0 mode: CR/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8dd expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787827204.818303:0:48444:0:(file.c:6959:ll_layout_lock_set()) ### file [0x240001b71:0x50:0x0](000000004296f63c) being reconfigured ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 3/1,0 mode: CR/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8dd expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827204.818333:0:48444:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x50:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1787827204.818336:0:48444:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 3/1,0 mode: CR/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da8dd expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827204.818342:0:48444:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 2/0,0 mode: CR/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da8dd expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1787827204.818359:0:48444:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827204.819284:0:48444:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x50:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1787827204.819297:0:48444:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f1:[0x240001b71:0x50:0x0](ffff8ab3f11fc2c8) current size 0, valid attrs 100201f0, mode 8ab3, uid -2015216048, gid -18322, new size 18446744072032287255, atime 1787827204.000000000, mtime 1787827204.000000000, ctime 1787827204.000000000, rc 0 00000080:00200008:2.0:1787827204.819329:0:48444:0:(file.c:479:ll_file_release()) START file f1:[0x240001b71:0x50:0x0](ffff8ab3f11fc2c8), flags 104001 00000002:00100000:2.0:1787827204.819371:0:48444:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3d0645e00 x1874671280571264/t17179869531(17179869531) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 576/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787827204.819383:0:48444:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787827204.819389:0:48444:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a27c00 x1874671280572032/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787827204.819404:0:48444:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a27c00 pname:cluuid:pid:xid:nid:opc:job touch:27aedc06-3022-4c11-aafb-6bd27e69b836:48444:1874671280572032:192.168.203.124@tcp:35:touch.0 00000100:00100000:2.0:1787827204.819442:0:48444:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913000 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.820280:0:48444:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a27c00 pname:cluuid:pid:xid:nid:opc:job touch:27aedc06-3022-4c11-aafb-6bd27e69b836:48444:1874671280572032:192.168.203.124@tcp:35:touch.0 00000002:00100000:2.0:1787827204.820297:0:48444:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3d0645e00 x1874671280571264/t17179869531(17179869531) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 576/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787827204.823507:0:48445:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.823529:0:48445:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.823546:0:48445:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.823561:0:48445:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.823569:0:48445:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.823577:0:48445:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.823582:0:48445:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.823589:0:48445:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e8bbf8 00010000:00010000:2.0:1787827204.823594:0:48445:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.823603:0:48445:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.823619:0:48445:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.823627:0:48445:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=25 00000080:00200000:2.0:1787827204.823634:0:48445:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.823639:0:48445:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1787827204.823643:0:48445:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.823647:0:48445:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d_f6.sanity-scrub, flags=25 00000080:00200000:2.0:1787827204.823652:0:48445:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x4f:0x0](ffff8ab3f11fa1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.823660:0:48445:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x4f:0x0](ffff8ab3f11fa1c8), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.823664:0:48445:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f2, dir=[0x240001b71:0x4f:0x0](ffff8ab3f11fa1c8), file ffff8ab3c393ea80, open_flags 8241, mode 81b6 opened 0 00000080:00200000:2.0:1787827204.823670:0:48445:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f2, dir=[0x240001b71:0x4f:0x0](ffff8ab3f11fa1c8), intent=open|creat 00000002:00010000:2.0:1787827204.823689:0:48445:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f2,[0x240001b71:0x51:0x0]) in obj [0x240001b71:0x4f:0x0], intent: open|creat flags 0101102 00010000:00010000:2.0:1787827204.823730:0:48445:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x4f:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 48445 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.823736:0:48445:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x4f:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48445 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1787827204.823744:0:48445:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1787827204.823752:0:48445:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a27840 x1874671280572288/t0(0) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1787827204.823764:0:48445:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 3/0,1 mode: --/CW res: [0x240001b71:0x4f:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 48445 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827204.823777:0:48445:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a27840 pname:cluuid:pid:xid:nid:opc:job dd:27aedc06-3022-4c11-aafb-6bd27e69b836:48445:1874671280572288:192.168.203.124@tcp:101:dd.0 00000100:00100000:2.0:1787827204.823879:0:48445:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c7913000 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.825473:0:48445:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a27840 pname:cluuid:pid:xid:nid:opc:job dd:27aedc06-3022-4c11-aafb-6bd27e69b836:48445:1874671280572288:192.168.203.124@tcp:101:dd.0 00010000:00010000:2.0:1787827204.825490:0:48445:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 4/0,1 mode: --/CW res: [0x240001b71:0x4f:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.825508:0:48445:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x51:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.825518:0:48445:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 4/0,1 mode: --/PR res: [0x240001b71:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.825530:0:48445:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.825542:0:48445:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 4/0,1 mode: PR/PR res: [0x240001b71:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.825555:0:48445:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.825564:0:48445:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 5/1,1 mode: PR/PR res: [0x240001b71:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.825581:0:48445:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 4/1,0 mode: PR/PR res: [0x240001b71:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827204.825592:0:48445:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8ab3d0a27840 x1874671280572288/t17179869534(17179869534) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 648/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1787827204.825606:0:48445:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0a27840 x1874671280572288/t17179869534(17179869534) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 648/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827204.825621:0:48445:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827204.825634:0:48445:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.825645:0:48445:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8ab3d084f800 type 13 mode 4 res 9663683441/81 (0 0) 00000080:00200000:2.0:1787827204.825650:0:48445:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240001b71:0x4f:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827204.825659:0:48445:0:(namei.c:1628:ll_create_it()) VFS Op:name=f2, dir=[0x240001b71:0x4f:0x0](ffff8ab3f11fa1c8), intent=open|creat 00000080:00200000:2.0:1787827204.825686:0:48445:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x51:0x0](ffff8ab3f11fec08) 00000080:00200000:2.0:1787827204.825765:0:48445:0:(file.c:6854:ll_layout_conf()) [0x240001b71:0x51:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1787827204.825770:0:48445:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240001b71:0x51:0x0](ffff8ab3f11fec08) 00000080:00010000:2.0:1787827204.825775:0:48445:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240001b71:0x51:0x0](ffff8ab3f11fec08) for lock 0x268e1a602cc25c1 00000080:00010000:2.0:1787827204.825785:0:48445:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240001b71:0x51:0x0](ffff8ab3f11fec08) 00000080:00200008:2.0:1787827204.825795:0:48445:0:(file.c:1090:ll_file_open()) START file f2:[0x240001b71:0x51:0x0](ffff8ab3f11fec08), flags 101101 00000002:00100000:2.0:1787827204.825815:0:48445:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8ab3d0a27840 x1874671280572288/t17179869534(17179869534) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 576/664 e 0 to 0 dl 1787827220 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1787827204.825852:0:48445:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc25c1 from it ffff8ab3d06869c0 00010000:00010000:2.0:1787827204.825858:0:48445:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 3/1,0 mode: PR/PR res: [0x240001b71:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.825867:0:48445:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827204.825923:0:48445:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x51:0x0](ffff8ab3f11fec08), inode mode 81a4 mask 40 00000080:00200008:2.0:1787827204.828231:0:48445:0:(file.c:2652:do_file_write_iter()) START file f2:[0x240001b71:0x51:0x0], ppos: 0, count: 1048576 00000080:00200000:2.0:1787827204.828244:0:48445:0:(file.c:2090:ll_file_io_generic()) f2: write ppos: 0, bytes: 1048576 00000080:00200000:2.0:1787827204.828249:0:48445:0:(file.c:1848:ll_io_set_mirror()) f2: desiginated mirror: 0 00000020:00200000:2.0:1787827204.828258:0:48445:0:(cl_io.c:218:cl_io_rw_init()) header@00000000105e2f2d[0x110, 1, [0x240001b71:0x51:0x0] hash] 00000020:00200000:2.0:1787827204.828261:0:48445:0:(cl_io.c:218:cl_io_rw_init()) io range: 2 [0, 1048576) 0 0 00000080:00200000:2.0:1787827204.828266:0:48445:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x51:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787827204.828280:0:48445:0:(file.c:2170:ll_file_io_generic()) Range lock [0, 255] 00020000:00200000:2.0:1787827204.828288:0:48445:0:(lov_io.c:1445:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1048576) 1048576, 1048576 00020000:00200000:2.0:1787827204.828291:0:48445:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827204.828306:0:48445:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 255] 00000008:00010000:2.0:1787827204.828321:0:48445:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1787827204.828326:0:48445:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8ab3d068eeb0, osc lock ffff8ab3c6ab1728, flags 0x0 00010000:00010000:2.0:1787827204.828365:0:48445:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.828373:0:48445:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->1048575] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.828381:0:48445:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->1048575] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827204.828394:0:48445:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a24b40 pname:cluuid:pid:xid:nid:opc:job dd:27aedc06-3022-4c11-aafb-6bd27e69b836:48445:1874671280572416:192.168.203.124@tcp:101:dd.0 00000100:00100000:2.0:1787827204.828444:0:48445:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79139c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.829357:0:48445:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a24b40 pname:cluuid:pid:xid:nid:opc:job dd:27aedc06-3022-4c11-aafb-6bd27e69b836:48445:1874671280572416:192.168.203.124@tcp:101:dd.0 00010000:00010000:2.0:1787827204.829373:0:48445:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.829390:0:48445:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.829404:0:48445:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.829417:0:48445:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.829430:0:48445:0:(osc_lock.c:192:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.829442:0:48445:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.829452:0:48445:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.829474:0:48445:0:(vvp_io.c:1339:vvp_io_write_start()) f2: write [0, 1048576) append 0 append lockpos 0 00000080:00200000:2.0:1787827204.829490:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.829532:0:48445:0:(rw26.c:875:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1787827204.829544:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000a9507cc0[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 0] 00000080:00200000:2.0:1787827204.829549:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000bc16d662 17ffffc0001001 3:0 ffff8ab3c6a9ce80 0 lru 00000080:00200000:2.0:1787827204.829558:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000885190a5 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.829561:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000a9507cc0 00000080:00200000:2.0:1787827204.829562:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1787827204.829570:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.829598:0:48445:0:(rw26.c:875:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1787827204.829604:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000e0795ff9[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 1] 00000080:00200000:2.0:1787827204.829608:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000006c982c36 17ffffc0001001 3:0 ffff8ab3c6a9cae0 1 lru 00000080:00200000:2.0:1787827204.829614:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000006383ca8 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.829617:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000e0795ff9 00000080:00200000:2.0:1787827204.829618:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1787827204.829621:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.829638:0:48445:0:(rw26.c:875:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1787827204.829643:0:48445:0:(rw26.c:933:ll_write_end()) page@000000003449af30[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 2] 00000080:00200000:2.0:1787827204.829647:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000013b54efe 17ffffc0001001 3:0 ffff8ab3c6a9cbc8 2 lru 00000080:00200000:2.0:1787827204.829654:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000884246e8 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.829656:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000003449af30 00000080:00200000:2.0:1787827204.829657:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1787827204.829659:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.829683:0:48445:0:(rw26.c:875:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1787827204.829688:0:48445:0:(rw26.c:933:ll_write_end()) page@000000001768a983[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 3] 00000080:00200000:2.0:1787827204.829691:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000002fddabcb 17ffffc0001001 3:0 ffff8ab3c6a9c910 3 lru 00000080:00200000:2.0:1787827204.829698:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000332614c0 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.829701:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000001768a983 00000080:00200000:2.0:1787827204.829702:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1787827204.829705:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.829722:0:48445:0:(rw26.c:875:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1787827204.829727:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000f2ca8a46[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 4] 00000080:00200000:2.0:1787827204.829730:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000073562f1c 17ffffc0001001 3:0 ffff8ab3c6a9c828 4 lru 00000080:00200000:2.0:1787827204.829737:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000009480a4dd 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.829739:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000f2ca8a46 00000080:00200000:2.0:1787827204.829741:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1787827204.829743:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.829760:0:48445:0:(rw26.c:875:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1787827204.829766:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000c6cc4509[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 5] 00000080:00200000:2.0:1787827204.829769:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000004533b8da 17ffffc0001001 3:0 ffff8ab3c6a9ccb0 5 lru 00000080:00200000:2.0:1787827204.829775:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000cd5afde9 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.829778:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000c6cc4509 00000080:00200000:2.0:1787827204.829779:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1787827204.829782:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.829798:0:48445:0:(rw26.c:875:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1787827204.829804:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000935a5a4d[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 6] 00000080:00200000:2.0:1787827204.829807:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000a4b0d74a 17ffffc0001001 3:0 ffff8ab3c6a9cd98 6 lru 00000080:00200000:2.0:1787827204.829814:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000bd3837d5 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.829817:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000935a5a4d 00000080:00200000:2.0:1787827204.829818:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1787827204.829820:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.829846:0:48445:0:(rw26.c:875:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:2.0:1787827204.829852:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000a1fc08c0[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 7] 00000080:00200000:2.0:1787827204.829856:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000008ed670b0 17ffffc0001001 3:0 ffff8ab3c6a9c2b8 7 lru 00000080:00200000:2.0:1787827204.829862:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000023e7d0aa 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.829865:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000a1fc08c0 00000080:00200000:2.0:1787827204.829866:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1787827204.829869:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.829917:0:48445:0:(rw26.c:875:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:2.0:1787827204.829924:0:48445:0:(rw26.c:933:ll_write_end()) page@000000007072af62[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 8] 00000080:00200000:2.0:1787827204.829928:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000638e70fa 17ffffc0001001 3:0 ffff8ab3d0bba1d0 8 lru 00000080:00200000:2.0:1787827204.829935:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000042db5386 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.829938:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000007072af62 00000080:00200000:2.0:1787827204.829940:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1787827204.829943:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.829967:0:48445:0:(rw26.c:875:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:2.0:1787827204.829973:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000a37cdede[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 9] 00000080:00200000:2.0:1787827204.829976:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000088414f8e 17ffffc0001021 2:0 ffff8ab3d0bba658 9 lru 00000080:00200000:2.0:1787827204.829984:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000086530db4 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.829987:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000a37cdede 00000080:00200000:2.0:1787827204.829988:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1787827204.829991:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830009:0:48445:0:(rw26.c:875:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830015:0:48445:0:(rw26.c:933:ll_write_end()) page@000000005b07271a[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 10] 00000080:00200000:2.0:1787827204.830018:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000008843d29d 17ffffc0001001 3:0 ffff8ab3d0bba000 10 lru 00000080:00200000:2.0:1787827204.830024:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000002c0a8253 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830027:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000005b07271a 00000080:00200000:2.0:1787827204.830028:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1787827204.830030:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830047:0:48445:0:(rw26.c:875:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830054:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000019da7271[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 11] 00000080:00200000:2.0:1787827204.830057:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000002a5a8f7d 17ffffc0001001 3:0 ffff8ab3d0bbae80 11 lru 00000080:00200000:2.0:1787827204.830064:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000c09106b7 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830066:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000019da7271 00000080:00200000:2.0:1787827204.830067:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1787827204.830069:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830088:0:48445:0:(rw26.c:875:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830108:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000036b50993[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 12] 00000080:00200000:2.0:1787827204.830111:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000d43524dd 17ffffc0001001 3:0 ffff8ab3d0bbaae0 12 lru 00000080:00200000:2.0:1787827204.830118:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000001d6868e5 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830120:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000036b50993 00000080:00200000:2.0:1787827204.830122:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1787827204.830125:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830143:0:48445:0:(rw26.c:875:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830148:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000016c4fe01[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 13] 00000080:00200000:2.0:1787827204.830151:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000017d312db 17ffffc0001001 3:0 ffff8ab3d0bbabc8 13 lru 00000080:00200000:2.0:1787827204.830158:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000001ab81e03 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830160:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000016c4fe01 00000080:00200000:2.0:1787827204.830162:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1787827204.830164:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830180:0:48445:0:(rw26.c:875:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830185:0:48445:0:(rw26.c:933:ll_write_end()) page@000000009f9f6a5f[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 14] 00000080:00200000:2.0:1787827204.830189:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000006c546c3 17ffffc0001001 3:0 ffff8ab3d0bba910 14 lru 00000080:00200000:2.0:1787827204.830195:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000558b976f 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830198:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000009f9f6a5f 00000080:00200000:2.0:1787827204.830199:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1787827204.830201:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830220:0:48445:0:(rw26.c:875:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830226:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000acdeb16e[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 15] 00000080:00200000:2.0:1787827204.830229:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000f7e2c3b3 17ffffc0001001 3:0 ffff8ab3d0bba828 15 lru 00000080:00200000:2.0:1787827204.830236:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000fb2dd9f0 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830238:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000acdeb16e 00000080:00200000:2.0:1787827204.830239:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1787827204.830242:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830260:0:48445:0:(rw26.c:875:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830265:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000ab4e8b20[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 16] 00000080:00200000:2.0:1787827204.830269:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000b95b7499 17ffffc0001001 3:0 ffff8ab3d0bbacb0 16 lru 00000080:00200000:2.0:1787827204.830275:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000000bacdcfd 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830278:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000ab4e8b20 00000080:00200000:2.0:1787827204.830279:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1787827204.830281:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830299:0:48445:0:(rw26.c:875:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830304:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000062d1eaad[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 17] 00000080:00200000:2.0:1787827204.830307:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000003a29dbb4 17ffffc0001001 3:0 ffff8ab3d0bbad98 17 lru 00000080:00200000:2.0:1787827204.830313:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000cc5bff49 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830316:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000062d1eaad 00000080:00200000:2.0:1787827204.830317:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1787827204.830320:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830337:0:48445:0:(rw26.c:875:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830342:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000c7506ddf[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 18] 00000080:00200000:2.0:1787827204.830345:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000005d4ba4bb 17ffffc0001001 3:0 ffff8ab3d0bba2b8 18 lru 00000080:00200000:2.0:1787827204.830352:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000093bb962c 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830354:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000c7506ddf 00000080:00200000:2.0:1787827204.830356:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1787827204.830358:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830376:0:48445:0:(rw26.c:875:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830381:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000e7c03a3a[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 19] 00000080:00200000:2.0:1787827204.830384:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000bf0dddce 17ffffc0001001 3:0 ffff8ab3d0bba488 19 lru 00000080:00200000:2.0:1787827204.830391:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000a2779a86 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830394:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000e7c03a3a 00000080:00200000:2.0:1787827204.830395:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 20. 00000080:00200000:2.0:1787827204.830397:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830414:0:48445:0:(rw26.c:875:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830419:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000c9455f7f[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 20] 00000080:00200000:2.0:1787827204.830422:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000de198bcd 17ffffc0001001 3:0 ffff8ab3d0bba3a0 20 lru 00000080:00200000:2.0:1787827204.830429:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000004bb7bae2 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830431:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000c9455f7f 00000080:00200000:2.0:1787827204.830432:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1787827204.830435:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830451:0:48445:0:(rw26.c:875:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830456:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000380fbbb8[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 21] 00000080:00200000:2.0:1787827204.830460:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000002c9b134b 17ffffc0001001 3:0 ffff8ab3d0bba9f8 21 lru 00000080:00200000:2.0:1787827204.830465:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000005c4b22da 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830468:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000380fbbb8 00000080:00200000:2.0:1787827204.830470:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 22. 00000080:00200000:2.0:1787827204.830472:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830491:0:48445:0:(rw26.c:875:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830496:0:48445:0:(rw26.c:933:ll_write_end()) page@000000003d719c7f[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 22] 00000080:00200000:2.0:1787827204.830499:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000592f5d8f 17ffffc0001001 3:0 ffff8ab3d0bba0e8 22 lru 00000080:00200000:2.0:1787827204.830505:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000593540ba 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830508:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000003d719c7f 00000080:00200000:2.0:1787827204.830509:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 23. 00000080:00200000:2.0:1787827204.830512:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830527:0:48445:0:(rw26.c:875:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830532:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000622afd33[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 23] 00000080:00200000:2.0:1787827204.830536:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000002e70bdec 17ffffc0001001 3:0 ffff8ab3d0bba740 23 lru 00000080:00200000:2.0:1787827204.830542:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000009e04e766 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830545:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000622afd33 00000080:00200000:2.0:1787827204.830546:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 24. 00000080:00200000:2.0:1787827204.830548:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830568:0:48445:0:(rw26.c:875:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830574:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000fb2112b4[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 24] 00000080:00200000:2.0:1787827204.830577:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000281fe467 17ffffc0001021 2:0 ffff8ab3d0bba570 24 lru 00000080:00200000:2.0:1787827204.830583:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000f66d53ba 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830586:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000fb2112b4 00000080:00200000:2.0:1787827204.830587:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 25. 00000080:00200000:2.0:1787827204.830589:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830613:0:48445:0:(rw26.c:875:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830619:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000d0526b34[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 25] 00000080:00200000:2.0:1787827204.830622:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000082aaac1e 17ffffc0001001 3:0 ffff8ab3f0665658 25 lru 00000080:00200000:2.0:1787827204.830628:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000001ca31d22 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830630:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000d0526b34 00000080:00200000:2.0:1787827204.830632:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 26. 00000080:00200000:2.0:1787827204.830634:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830648:0:48445:0:(rw26.c:875:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830653:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000e4140a05[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 26] 00000080:00200000:2.0:1787827204.830656:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000e1754131 17ffffc0001001 3:0 ffff8ab3f0665000 26 lru 00000080:00200000:2.0:1787827204.830662:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000e0c62b24 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830665:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000e4140a05 00000080:00200000:2.0:1787827204.830667:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 27. 00000080:00200000:2.0:1787827204.830669:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830683:0:48445:0:(rw26.c:875:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830689:0:48445:0:(rw26.c:933:ll_write_end()) page@000000008093ea41[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 27] 00000080:00200000:2.0:1787827204.830692:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000c100a82e 17ffffc0001001 3:0 ffff8ab3f0665e80 27 lru 00000080:00200000:2.0:1787827204.830698:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000065b6301e 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830701:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000008093ea41 00000080:00200000:2.0:1787827204.830703:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 28. 00000080:00200000:2.0:1787827204.830705:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830719:0:48445:0:(rw26.c:875:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830725:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000afeb09a7[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 28] 00000080:00200000:2.0:1787827204.830728:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000e0a3cb20 17ffffc0001001 3:0 ffff8ab3f0665ae0 28 lru 00000080:00200000:2.0:1787827204.830735:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000006dd47278 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830737:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000afeb09a7 00000080:00200000:2.0:1787827204.830738:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 29. 00000080:00200000:2.0:1787827204.830740:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830758:0:48445:0:(rw26.c:875:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830763:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000119ae25e[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 29] 00000080:00200000:2.0:1787827204.830766:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000061769815 17ffffc0001001 3:0 ffff8ab3f0665bc8 29 lru 00000080:00200000:2.0:1787827204.830773:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000099536c04 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830775:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000119ae25e 00000080:00200000:2.0:1787827204.830777:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 30. 00000080:00200000:2.0:1787827204.830779:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830793:0:48445:0:(rw26.c:875:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830798:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000f2a28301[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 30] 00000080:00200000:2.0:1787827204.830801:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000003860fd06 17ffffc0001001 3:0 ffff8ab3f0665910 30 lru 00000080:00200000:2.0:1787827204.830807:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000038b9b510 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830810:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000f2a28301 00000080:00200000:2.0:1787827204.830812:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 31. 00000080:00200000:2.0:1787827204.830813:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830827:0:48445:0:(rw26.c:875:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830831:0:48445:0:(rw26.c:933:ll_write_end()) page@000000003793ed40[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 31] 00000080:00200000:2.0:1787827204.830835:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000005ec958b8 17ffffc0001001 3:0 ffff8ab3f0665828 31 lru 00000080:00200000:2.0:1787827204.830842:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000002f45fa2c 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830845:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000003793ed40 00000080:00200000:2.0:1787827204.830847:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 32. 00000080:00200000:2.0:1787827204.830849:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.830866:0:48445:0:(rw26.c:875:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:2.0:1787827204.830871:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000a94146a2[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 32] 00000080:00200000:2.0:1787827204.830875:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000145de4e8 17ffffc0001001 3:0 ffff8ab3f0665cb0 32 lru 00000080:00200000:2.0:1787827204.830880:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000cb0eac8b 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.830883:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000a94146a2 00000080:00200000:2.0:1787827204.831059:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 33. 00000080:00200000:2.0:1787827204.831062:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831081:0:48445:0:(rw26.c:875:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831086:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000734899ce[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 33] 00000080:00200000:2.0:1787827204.831090:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000e158a6aa 17ffffc0001001 3:0 ffff8ab3f0665d98 33 lru 00000080:00200000:2.0:1787827204.831110:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000050e9b870 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831113:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000734899ce 00000080:00200000:2.0:1787827204.831114:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 34. 00000080:00200000:2.0:1787827204.831117:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 34 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831135:0:48445:0:(rw26.c:875:ll_write_end()) pos 139264, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831140:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000012053a92[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 34] 00000080:00200000:2.0:1787827204.831143:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000ca96338a 17ffffc0001001 3:0 ffff8ab3f06652b8 34 lru 00000080:00200000:2.0:1787827204.831149:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000067d84120 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831152:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000012053a92 00000080:00200000:2.0:1787827204.831153:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 35. 00000080:00200000:2.0:1787827204.831155:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 35 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831171:0:48445:0:(rw26.c:875:ll_write_end()) pos 143360, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831176:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000069000111[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 35] 00000080:00200000:2.0:1787827204.831180:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000f4de0292 17ffffc0001001 3:0 ffff8ab3f0665488 35 lru 00000080:00200000:2.0:1787827204.831186:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000398d73f8 35: 1< 0 - - > 2< 143360 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831188:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000069000111 00000080:00200000:2.0:1787827204.831190:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 36. 00000080:00200000:2.0:1787827204.831192:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 36 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831207:0:48445:0:(rw26.c:875:ll_write_end()) pos 147456, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831211:0:48445:0:(rw26.c:933:ll_write_end()) page@000000009e92003e[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 36] 00000080:00200000:2.0:1787827204.831215:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000009e4d90c6 17ffffc0001001 3:0 ffff8ab3f06653a0 36 lru 00000080:00200000:2.0:1787827204.831221:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000fe5b5ca8 36: 1< 0 - - > 2< 147456 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831224:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000009e92003e 00000080:00200000:2.0:1787827204.831225:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 37. 00000080:00200000:2.0:1787827204.831227:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 37 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831244:0:48445:0:(rw26.c:875:ll_write_end()) pos 151552, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831249:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000a028f1e3[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 37] 00000080:00200000:2.0:1787827204.831253:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000897bb9c3 17ffffc0001001 3:0 ffff8ab3f06659f8 37 lru 00000080:00200000:2.0:1787827204.831259:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000aa57d125 37: 1< 0 - - > 2< 151552 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831261:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000a028f1e3 00000080:00200000:2.0:1787827204.831263:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 38. 00000080:00200000:2.0:1787827204.831265:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 38 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831282:0:48445:0:(rw26.c:875:ll_write_end()) pos 155648, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831287:0:48445:0:(rw26.c:933:ll_write_end()) page@000000008c2aeb39[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 38] 00000080:00200000:2.0:1787827204.831290:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000009eca5c41 17ffffc0001001 3:0 ffff8ab3f06650e8 38 lru 00000080:00200000:2.0:1787827204.831296:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000000dc94fb8 38: 1< 0 - - > 2< 155648 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831298:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000008c2aeb39 00000080:00200000:2.0:1787827204.831300:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 39. 00000080:00200000:2.0:1787827204.831302:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 39 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831323:0:48445:0:(rw26.c:875:ll_write_end()) pos 159744, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831328:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000f16b69d0[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 39] 00000080:00200000:2.0:1787827204.831332:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000000141734b 17ffffc0001021 2:0 ffff8ab3f0665740 39 lru 00000080:00200000:2.0:1787827204.831339:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000575382d8 39: 1< 0 - - > 2< 159744 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831341:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000f16b69d0 00000080:00200000:2.0:1787827204.831343:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 40. 00000080:00200000:2.0:1787827204.831345:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 40 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831360:0:48445:0:(rw26.c:875:ll_write_end()) pos 163840, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831365:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000b85c99a9[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 40] 00000080:00200000:2.0:1787827204.831368:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000000390396a 17ffffc0001001 3:0 ffff8ab3f0665570 40 lru 00000080:00200000:2.0:1787827204.831375:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000eee8b4be 40: 1< 0 - - > 2< 163840 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831377:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000b85c99a9 00000080:00200000:2.0:1787827204.831379:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 41. 00000080:00200000:2.0:1787827204.831381:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 41 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831394:0:48445:0:(rw26.c:875:ll_write_end()) pos 167936, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831399:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000059aa565a[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 41] 00000080:00200000:2.0:1787827204.831402:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000018a42345 17ffffc0001001 3:0 ffff8ab3f06651d0 41 lru 00000080:00200000:2.0:1787827204.831408:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000000c973cbc 41: 1< 0 - - > 2< 167936 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831411:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000059aa565a 00000080:00200000:2.0:1787827204.831412:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 42. 00000080:00200000:2.0:1787827204.831414:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 42 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831436:0:48445:0:(rw26.c:875:ll_write_end()) pos 172032, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831441:0:48445:0:(rw26.c:933:ll_write_end()) page@000000005db80fcf[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 42] 00000080:00200000:2.0:1787827204.831445:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000984c9943 17ffffc0001001 3:0 ffff8ab3c49291d0 42 lru 00000080:00200000:2.0:1787827204.831451:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000002614b78e 42: 1< 0 - - > 2< 172032 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831454:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000005db80fcf 00000080:00200000:2.0:1787827204.831455:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 43. 00000080:00200000:2.0:1787827204.831457:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 43 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831472:0:48445:0:(rw26.c:875:ll_write_end()) pos 176128, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831476:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000082365753[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 43] 00000080:00200000:2.0:1787827204.831480:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000031a3767a 17ffffc0001001 3:0 ffff8ab3c4929658 43 lru 00000080:00200000:2.0:1787827204.831486:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000007328dc74 43: 1< 0 - - > 2< 176128 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831489:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000082365753 00000080:00200000:2.0:1787827204.831490:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 44. 00000080:00200000:2.0:1787827204.831492:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 44 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831508:0:48445:0:(rw26.c:875:ll_write_end()) pos 180224, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831513:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000007d32888[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 44] 00000080:00200000:2.0:1787827204.831516:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000017f2b91a 17ffffc0001001 3:0 ffff8ab3c4929000 44 lru 00000080:00200000:2.0:1787827204.831521:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000691a8e97 44: 1< 0 - - > 2< 180224 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831523:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000007d32888 00000080:00200000:2.0:1787827204.831525:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 45. 00000080:00200000:2.0:1787827204.831527:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 45 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831542:0:48445:0:(rw26.c:875:ll_write_end()) pos 184320, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831547:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000c7be9c77[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 45] 00000080:00200000:2.0:1787827204.831550:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000004904a54b 17ffffc0001001 3:0 ffff8ab3c4929e80 45 lru 00000080:00200000:2.0:1787827204.831557:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000b6f054e0 45: 1< 0 - - > 2< 184320 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831560:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000c7be9c77 00000080:00200000:2.0:1787827204.831561:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 46. 00000080:00200000:2.0:1787827204.831563:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 46 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831576:0:48445:0:(rw26.c:875:ll_write_end()) pos 188416, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831581:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000b9449f59[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 46] 00000080:00200000:2.0:1787827204.831585:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000a0266238 17ffffc0001001 3:0 ffff8ab3c4929ae0 46 lru 00000080:00200000:2.0:1787827204.831591:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000009c01905e 46: 1< 0 - - > 2< 188416 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831594:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000b9449f59 00000080:00200000:2.0:1787827204.831595:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 47. 00000080:00200000:2.0:1787827204.831597:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 47 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831612:0:48445:0:(rw26.c:875:ll_write_end()) pos 192512, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831617:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000b518bc4a[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 47] 00000080:00200000:2.0:1787827204.831620:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000005d89964b 17ffffc0001001 3:0 ffff8ab3c4929bc8 47 lru 00000080:00200000:2.0:1787827204.831626:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000bb17a923 47: 1< 0 - - > 2< 192512 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831629:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000b518bc4a 00000080:00200000:2.0:1787827204.831631:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 48. 00000080:00200000:2.0:1787827204.831633:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 48 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831648:0:48445:0:(rw26.c:875:ll_write_end()) pos 196608, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831653:0:48445:0:(rw26.c:933:ll_write_end()) page@000000008ab1d8e7[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 48] 00000080:00200000:2.0:1787827204.831657:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000007a38e007 17ffffc0001001 3:0 ffff8ab3c4929910 48 lru 00000080:00200000:2.0:1787827204.831663:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000119c1d80 48: 1< 0 - - > 2< 196608 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831665:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000008ab1d8e7 00000080:00200000:2.0:1787827204.831666:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 49. 00000080:00200000:2.0:1787827204.831669:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 49 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831684:0:48445:0:(rw26.c:875:ll_write_end()) pos 200704, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831690:0:48445:0:(rw26.c:933:ll_write_end()) page@000000009e34aff4[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 49] 00000080:00200000:2.0:1787827204.831693:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000001d813cf2 17ffffc0001001 3:0 ffff8ab3c4929828 49 lru 00000080:00200000:2.0:1787827204.831699:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000052c230a5 49: 1< 0 - - > 2< 200704 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831702:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000009e34aff4 00000080:00200000:2.0:1787827204.831704:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 50. 00000080:00200000:2.0:1787827204.831706:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 50 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831720:0:48445:0:(rw26.c:875:ll_write_end()) pos 204800, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831726:0:48445:0:(rw26.c:933:ll_write_end()) page@000000003151a122[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 50] 00000080:00200000:2.0:1787827204.831729:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000941b3a5f 17ffffc0001001 3:0 ffff8ab3c4929cb0 50 lru 00000080:00200000:2.0:1787827204.831736:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000001c832560 50: 1< 0 - - > 2< 204800 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831738:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000003151a122 00000080:00200000:2.0:1787827204.831740:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 51. 00000080:00200000:2.0:1787827204.831742:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 51 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831757:0:48445:0:(rw26.c:875:ll_write_end()) pos 208896, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831763:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000926f2b53[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 51] 00000080:00200000:2.0:1787827204.831766:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000e338313b 17ffffc0001001 3:0 ffff8ab3c4929d98 51 lru 00000080:00200000:2.0:1787827204.831773:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000017ab5ea0 51: 1< 0 - - > 2< 208896 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831776:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000926f2b53 00000080:00200000:2.0:1787827204.831778:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 52. 00000080:00200000:2.0:1787827204.831780:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 52 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831797:0:48445:0:(rw26.c:875:ll_write_end()) pos 212992, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831802:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000038833083[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 52] 00000080:00200000:2.0:1787827204.831806:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000502b7a3c 17ffffc0001001 3:0 ffff8ab3c49292b8 52 lru 00000080:00200000:2.0:1787827204.831812:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000403a64d0 52: 1< 0 - - > 2< 212992 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831814:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000038833083 00000080:00200000:2.0:1787827204.831815:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 53. 00000080:00200000:2.0:1787827204.831818:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 53 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831834:0:48445:0:(rw26.c:875:ll_write_end()) pos 217088, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831840:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000f13771b6[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 53] 00000080:00200000:2.0:1787827204.831843:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000c5a9b79d 17ffffc0001001 3:0 ffff8ab3c4929488 53 lru 00000080:00200000:2.0:1787827204.831849:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000afb1cc74 53: 1< 0 - - > 2< 217088 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831851:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000f13771b6 00000080:00200000:2.0:1787827204.831853:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 54. 00000080:00200000:2.0:1787827204.831855:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 54 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831876:0:48445:0:(rw26.c:875:ll_write_end()) pos 221184, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831882:0:48445:0:(rw26.c:933:ll_write_end()) page@000000000659b7f6[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 54] 00000080:00200000:2.0:1787827204.831903:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000002e56bf35 17ffffc0001021 2:0 ffff8ab3c49293a0 54 lru 00000080:00200000:2.0:1787827204.831910:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000004184c09a 54: 1< 0 - - > 2< 221184 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831913:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000000659b7f6 00000080:00200000:2.0:1787827204.831914:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 55. 00000080:00200000:2.0:1787827204.831917:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 55 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831935:0:48445:0:(rw26.c:875:ll_write_end()) pos 225280, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831940:0:48445:0:(rw26.c:933:ll_write_end()) page@000000005bafce7a[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 55] 00000080:00200000:2.0:1787827204.831943:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000afd4e026 17ffffc0001001 3:0 ffff8ab3c49299f8 55 lru 00000080:00200000:2.0:1787827204.831949:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000007cf174cc 55: 1< 0 - - > 2< 225280 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831951:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000005bafce7a 00000080:00200000:2.0:1787827204.831953:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 56. 00000080:00200000:2.0:1787827204.831955:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 56 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.831971:0:48445:0:(rw26.c:875:ll_write_end()) pos 229376, len 4096, copied 4096 00000080:00200000:2.0:1787827204.831976:0:48445:0:(rw26.c:933:ll_write_end()) page@000000002cf824b8[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 56] 00000080:00200000:2.0:1787827204.831979:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000003c19486a 17ffffc0001001 3:0 ffff8ab3c49290e8 56 lru 00000080:00200000:2.0:1787827204.831985:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000fb0003c2 56: 1< 0 - - > 2< 229376 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.831987:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000002cf824b8 00000080:00200000:2.0:1787827204.831989:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 57. 00000080:00200000:2.0:1787827204.831991:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 57 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832008:0:48445:0:(rw26.c:875:ll_write_end()) pos 233472, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832013:0:48445:0:(rw26.c:933:ll_write_end()) page@000000006e00213f[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 57] 00000080:00200000:2.0:1787827204.832017:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000002badf734 17ffffc0001001 3:0 ffff8ab3c4929740 57 lru 00000080:00200000:2.0:1787827204.832023:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000ee4766c0 57: 1< 0 - - > 2< 233472 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832025:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000006e00213f 00000080:00200000:2.0:1787827204.832027:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 58. 00000080:00200000:2.0:1787827204.832029:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 58 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832046:0:48445:0:(rw26.c:875:ll_write_end()) pos 237568, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832051:0:48445:0:(rw26.c:933:ll_write_end()) page@000000002f5096fd[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 58] 00000080:00200000:2.0:1787827204.832054:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000000d1eebcd 17ffffc0001001 3:0 ffff8ab3c4929570 58 lru 00000080:00200000:2.0:1787827204.832061:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000870de608 58: 1< 0 - - > 2< 237568 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832064:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000002f5096fd 00000080:00200000:2.0:1787827204.832065:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 59. 00000080:00200000:2.0:1787827204.832068:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 59 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832107:0:48445:0:(rw26.c:875:ll_write_end()) pos 241664, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832114:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000015ebb2a2[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 59] 00000080:00200000:2.0:1787827204.832117:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000085413836 17ffffc0001001 3:0 ffff8ab3f075a000 59 lru 00000080:00200000:2.0:1787827204.832125:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000022378647 59: 1< 0 - - > 2< 241664 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832127:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000015ebb2a2 00000080:00200000:2.0:1787827204.832129:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 60. 00000080:00200000:2.0:1787827204.832131:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 60 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832148:0:48445:0:(rw26.c:875:ll_write_end()) pos 245760, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832153:0:48445:0:(rw26.c:933:ll_write_end()) page@000000007ff6f647[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 60] 00000080:00200000:2.0:1787827204.832157:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000006c5e3d2a 17ffffc0001001 3:0 ffff8ab3f075ae80 60 lru 00000080:00200000:2.0:1787827204.832163:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000073a48cfb 60: 1< 0 - - > 2< 245760 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832166:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000007ff6f647 00000080:00200000:2.0:1787827204.832167:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 61. 00000080:00200000:2.0:1787827204.832169:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 61 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832189:0:48445:0:(rw26.c:875:ll_write_end()) pos 249856, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832194:0:48445:0:(rw26.c:933:ll_write_end()) page@000000000268f790[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 61] 00000080:00200000:2.0:1787827204.832197:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000006f8430a 17ffffc0001001 3:0 ffff8ab3f075aae0 61 lru 00000080:00200000:2.0:1787827204.832204:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000048b6da2a 61: 1< 0 - - > 2< 249856 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832207:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000000268f790 00000080:00200000:2.0:1787827204.832208:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 62. 00000080:00200000:2.0:1787827204.832210:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 62 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832225:0:48445:0:(rw26.c:875:ll_write_end()) pos 253952, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832230:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000091a21790[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 62] 00000080:00200000:2.0:1787827204.832233:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000b6ee94ae 17ffffc0001001 3:0 ffff8ab3f075abc8 62 lru 00000080:00200000:2.0:1787827204.832240:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000001e3d782a 62: 1< 0 - - > 2< 253952 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832242:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000091a21790 00000080:00200000:2.0:1787827204.832244:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 63. 00000080:00200000:2.0:1787827204.832246:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 63 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832262:0:48445:0:(rw26.c:875:ll_write_end()) pos 258048, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832267:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000f2cf2acc[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 63] 00000080:00200000:2.0:1787827204.832271:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000040514ee5 17ffffc0001001 3:0 ffff8ab3f075a910 63 lru 00000080:00200000:2.0:1787827204.832278:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000000eb695c7 63: 1< 0 - - > 2< 258048 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832280:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000f2cf2acc 00000080:00200000:2.0:1787827204.832282:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 64. 00000080:00200000:2.0:1787827204.832284:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 64 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832304:0:48445:0:(rw26.c:875:ll_write_end()) pos 262144, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832309:0:48445:0:(rw26.c:933:ll_write_end()) page@000000005a170190[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 64] 00000080:00200000:2.0:1787827204.832313:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000c358df09 17ffffc0001001 3:0 ffff8ab3f075a828 64 lru 00000080:00200000:2.0:1787827204.832319:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000d3135be9 64: 1< 0 - - > 2< 262144 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832322:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000005a170190 00000080:00200000:2.0:1787827204.832323:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 65. 00000080:00200000:2.0:1787827204.832325:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 65 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832340:0:48445:0:(rw26.c:875:ll_write_end()) pos 266240, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832345:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000063fc21e3[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 65] 00000080:00200000:2.0:1787827204.832349:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000c2c262d7 17ffffc0001001 3:0 ffff8ab3f075acb0 65 lru 00000080:00200000:2.0:1787827204.832356:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000006037d258 65: 1< 0 - - > 2< 266240 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832359:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000063fc21e3 00000080:00200000:2.0:1787827204.832360:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 66. 00000080:00200000:2.0:1787827204.832362:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 66 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832378:0:48445:0:(rw26.c:875:ll_write_end()) pos 270336, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832383:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000aae11227[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 66] 00000080:00200000:2.0:1787827204.832387:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000000a5e4ec8 17ffffc0001001 3:0 ffff8ab3f075ad98 66 lru 00000080:00200000:2.0:1787827204.832393:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000004a8ff56 66: 1< 0 - - > 2< 270336 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832395:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000aae11227 00000080:00200000:2.0:1787827204.832397:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 67. 00000080:00200000:2.0:1787827204.832399:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 67 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832416:0:48445:0:(rw26.c:875:ll_write_end()) pos 274432, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832421:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000c937a62f[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 67] 00000080:00200000:2.0:1787827204.832424:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000d86fd9ea 17ffffc0001001 3:0 ffff8ab3f075a2b8 67 lru 00000080:00200000:2.0:1787827204.832431:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000001ba8dc41 67: 1< 0 - - > 2< 274432 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832433:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000c937a62f 00000080:00200000:2.0:1787827204.832435:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 68. 00000080:00200000:2.0:1787827204.832437:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 68 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832457:0:48445:0:(rw26.c:875:ll_write_end()) pos 278528, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832462:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000036eeff73[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 68] 00000080:00200000:2.0:1787827204.832465:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000004b4355e0 17ffffc0001001 3:0 ffff8ab3f075a488 68 lru 00000080:00200000:2.0:1787827204.832471:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000a65241a0 68: 1< 0 - - > 2< 278528 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832474:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000036eeff73 00000080:00200000:2.0:1787827204.832475:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 69. 00000080:00200000:2.0:1787827204.832477:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 69 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832498:0:48445:0:(rw26.c:875:ll_write_end()) pos 282624, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832504:0:48445:0:(rw26.c:933:ll_write_end()) page@000000007e310398[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 69] 00000080:00200000:2.0:1787827204.832507:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000009950fbb2 17ffffc0001021 2:0 ffff8ab3f075a3a0 69 lru 00000080:00200000:2.0:1787827204.832514:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000f4cfb2a8 69: 1< 0 - - > 2< 282624 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832516:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000007e310398 00000080:00200000:2.0:1787827204.832517:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 70. 00000080:00200000:2.0:1787827204.832520:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 70 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832537:0:48445:0:(rw26.c:875:ll_write_end()) pos 286720, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832542:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000381b34a1[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 70] 00000080:00200000:2.0:1787827204.832546:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000983cdf9d 17ffffc0001001 3:0 ffff8ab3f075a9f8 70 lru 00000080:00200000:2.0:1787827204.832553:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000007f61da3c 70: 1< 0 - - > 2< 286720 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832555:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000381b34a1 00000080:00200000:2.0:1787827204.832557:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 71. 00000080:00200000:2.0:1787827204.832559:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 71 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832575:0:48445:0:(rw26.c:875:ll_write_end()) pos 290816, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832580:0:48445:0:(rw26.c:933:ll_write_end()) page@000000000afa28d2[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 71] 00000080:00200000:2.0:1787827204.832583:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000f8a21229 17ffffc0001001 3:0 ffff8ab3f075a0e8 71 lru 00000080:00200000:2.0:1787827204.832590:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000f9f14277 71: 1< 0 - - > 2< 290816 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832593:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000000afa28d2 00000080:00200000:2.0:1787827204.832594:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 72. 00000080:00200000:2.0:1787827204.832596:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 72 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832611:0:48445:0:(rw26.c:875:ll_write_end()) pos 294912, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832616:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000b7d437c9[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 72] 00000080:00200000:2.0:1787827204.832620:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000753e02b7 17ffffc0001001 3:0 ffff8ab3f075a740 72 lru 00000080:00200000:2.0:1787827204.832626:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000007725b9de 72: 1< 0 - - > 2< 294912 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832628:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000b7d437c9 00000080:00200000:2.0:1787827204.832629:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 73. 00000080:00200000:2.0:1787827204.832631:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 73 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832647:0:48445:0:(rw26.c:875:ll_write_end()) pos 299008, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832652:0:48445:0:(rw26.c:933:ll_write_end()) page@000000008564cddb[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 73] 00000080:00200000:2.0:1787827204.832656:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000001ac21b7a 17ffffc0001001 3:0 ffff8ab3f075a570 73 lru 00000080:00200000:2.0:1787827204.832662:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000065afa999 73: 1< 0 - - > 2< 299008 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832665:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000008564cddb 00000080:00200000:2.0:1787827204.832666:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 74. 00000080:00200000:2.0:1787827204.832668:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 74 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832682:0:48445:0:(rw26.c:875:ll_write_end()) pos 303104, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832687:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000f5fe22cb[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 74] 00000080:00200000:2.0:1787827204.832691:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000000ca5b75a 17ffffc0001001 3:0 ffff8ab3f075a1d0 74 lru 00000080:00200000:2.0:1787827204.832697:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000c581ac6a 74: 1< 0 - - > 2< 303104 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832699:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000f5fe22cb 00000080:00200000:2.0:1787827204.832700:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 75. 00000080:00200000:2.0:1787827204.832703:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 75 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832721:0:48445:0:(rw26.c:875:ll_write_end()) pos 307200, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832725:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000436f8562[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 75] 00000080:00200000:2.0:1787827204.832729:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000082c9030a 17ffffc0001001 3:0 ffff8ab3f075a658 75 lru 00000080:00200000:2.0:1787827204.832735:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000580a86cd 75: 1< 0 - - > 2< 307200 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832737:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000436f8562 00000080:00200000:2.0:1787827204.832739:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 76. 00000080:00200000:2.0:1787827204.832741:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 76 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832764:0:48445:0:(rw26.c:875:ll_write_end()) pos 311296, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832769:0:48445:0:(rw26.c:933:ll_write_end()) page@000000006b32d923[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 76] 00000080:00200000:2.0:1787827204.832772:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000003a6107f0 17ffffc0001001 3:0 ffff8ab3f042f570 76 lru 00000080:00200000:2.0:1787827204.832778:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000f2a866aa 76: 1< 0 - - > 2< 311296 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832781:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000006b32d923 00000080:00200000:2.0:1787827204.832783:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 77. 00000080:00200000:2.0:1787827204.832785:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 77 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832798:0:48445:0:(rw26.c:875:ll_write_end()) pos 315392, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832803:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000067081bec[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 77] 00000080:00200000:2.0:1787827204.832806:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000955df8dc 17ffffc0001001 3:0 ffff8ab3f042f1d0 77 lru 00000080:00200000:2.0:1787827204.832812:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000078c4dc8e 77: 1< 0 - - > 2< 315392 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832814:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000067081bec 00000080:00200000:2.0:1787827204.832816:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 78. 00000080:00200000:2.0:1787827204.832818:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 78 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832829:0:48445:0:(rw26.c:875:ll_write_end()) pos 319488, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832834:0:48445:0:(rw26.c:933:ll_write_end()) page@000000000def0ffe[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 78] 00000080:00200000:2.0:1787827204.832837:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000005f379b1b 17ffffc0001001 3:0 ffff8ab3f042f658 78 lru 00000080:00200000:2.0:1787827204.832843:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000009fc2e361 78: 1< 0 - - > 2< 319488 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832846:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000000def0ffe 00000080:00200000:2.0:1787827204.832847:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 79. 00000080:00200000:2.0:1787827204.832849:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 79 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832862:0:48445:0:(rw26.c:875:ll_write_end()) pos 323584, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832866:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000f265a541[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 79] 00000080:00200000:2.0:1787827204.832870:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000862562dc 17ffffc0001001 3:0 ffff8ab3f042f000 79 lru 00000080:00200000:2.0:1787827204.832876:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000065df5ace 79: 1< 0 - - > 2< 323584 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832877:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000f265a541 00000080:00200000:2.0:1787827204.832879:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 80. 00000080:00200000:2.0:1787827204.832881:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 80 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832917:0:48445:0:(rw26.c:875:ll_write_end()) pos 327680, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832923:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000df80d64b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 80] 00000080:00200000:2.0:1787827204.832926:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000008fdb145c 17ffffc0001001 3:0 ffff8ab3f042fe80 80 lru 00000080:00200000:2.0:1787827204.832933:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000e5f50178 80: 1< 0 - - > 2< 327680 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832936:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000df80d64b 00000080:00200000:2.0:1787827204.832937:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 81. 00000080:00200000:2.0:1787827204.832940:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 81 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832954:0:48445:0:(rw26.c:875:ll_write_end()) pos 331776, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832958:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000eef88f1e[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 81] 00000080:00200000:2.0:1787827204.832962:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000083b9fc77 17ffffc0001001 3:0 ffff8ab3f042fae0 81 lru 00000080:00200000:2.0:1787827204.832969:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000ca09fb77 81: 1< 0 - - > 2< 331776 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.832971:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000eef88f1e 00000080:00200000:2.0:1787827204.832972:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 82. 00000080:00200000:2.0:1787827204.832975:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 82 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.832991:0:48445:0:(rw26.c:875:ll_write_end()) pos 335872, len 4096, copied 4096 00000080:00200000:2.0:1787827204.832996:0:48445:0:(rw26.c:933:ll_write_end()) page@000000006ed5be6f[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 82] 00000080:00200000:2.0:1787827204.832999:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000093a4a57 17ffffc0001001 3:0 ffff8ab3f042fbc8 82 lru 00000080:00200000:2.0:1787827204.833004:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000c74e9fe6 82: 1< 0 - - > 2< 335872 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833006:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000006ed5be6f 00000080:00200000:2.0:1787827204.833007:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 83. 00000080:00200000:2.0:1787827204.833009:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 83 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833022:0:48445:0:(rw26.c:875:ll_write_end()) pos 339968, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833026:0:48445:0:(rw26.c:933:ll_write_end()) page@000000007e10d3a3[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 83] 00000080:00200000:2.0:1787827204.833029:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000fdb5182e 17ffffc0001001 3:0 ffff8ab3f042f910 83 lru 00000080:00200000:2.0:1787827204.833036:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000006107cf3b 83: 1< 0 - - > 2< 339968 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833038:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000007e10d3a3 00000080:00200000:2.0:1787827204.833040:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 84. 00000080:00200000:2.0:1787827204.833042:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 84 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833059:0:48445:0:(rw26.c:875:ll_write_end()) pos 344064, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833064:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000033fa7ac2[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 84] 00000080:00200000:2.0:1787827204.833067:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000003fb6d101 17ffffc0001021 2:0 ffff8ab3f042f828 84 lru 00000080:00200000:2.0:1787827204.833072:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000001f4bee5a 84: 1< 0 - - > 2< 344064 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833074:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000033fa7ac2 00000080:00200000:2.0:1787827204.833075:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 85. 00000080:00200000:2.0:1787827204.833077:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 85 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833089:0:48445:0:(rw26.c:875:ll_write_end()) pos 348160, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833107:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000019a7e67b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 85] 00000080:00200000:2.0:1787827204.833111:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000032d984e3 17ffffc0001001 3:0 ffff8ab3f042fcb0 85 lru 00000080:00200000:2.0:1787827204.833117:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000009cb64b64 85: 1< 0 - - > 2< 348160 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833119:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000019a7e67b 00000080:00200000:2.0:1787827204.833120:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 86. 00000080:00200000:2.0:1787827204.833122:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 86 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833137:0:48445:0:(rw26.c:875:ll_write_end()) pos 352256, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833141:0:48445:0:(rw26.c:933:ll_write_end()) page@000000002444f89b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 86] 00000080:00200000:2.0:1787827204.833144:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000007bebf2a 17ffffc0001001 3:0 ffff8ab3f042fd98 86 lru 00000080:00200000:2.0:1787827204.833150:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000cc78d5ef 86: 1< 0 - - > 2< 352256 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833152:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000002444f89b 00000080:00200000:2.0:1787827204.833153:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 87. 00000080:00200000:2.0:1787827204.833155:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 87 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833167:0:48445:0:(rw26.c:875:ll_write_end()) pos 356352, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833171:0:48445:0:(rw26.c:933:ll_write_end()) page@000000009b11d0b1[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 87] 00000080:00200000:2.0:1787827204.833174:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000b1b4a1e8 17ffffc0001001 3:0 ffff8ab3f042f2b8 87 lru 00000080:00200000:2.0:1787827204.833180:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000003f0807ff 87: 1< 0 - - > 2< 356352 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833182:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000009b11d0b1 00000080:00200000:2.0:1787827204.833183:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 88. 00000080:00200000:2.0:1787827204.833185:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 88 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833196:0:48445:0:(rw26.c:875:ll_write_end()) pos 360448, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833200:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000f9193cc6[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 88] 00000080:00200000:2.0:1787827204.833203:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000356e29a4 17ffffc0001001 3:0 ffff8ab3f042f488 88 lru 00000080:00200000:2.0:1787827204.833209:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000087c4c2fe 88: 1< 0 - - > 2< 360448 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833211:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000f9193cc6 00000080:00200000:2.0:1787827204.833212:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 89. 00000080:00200000:2.0:1787827204.833214:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 89 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833228:0:48445:0:(rw26.c:875:ll_write_end()) pos 364544, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833232:0:48445:0:(rw26.c:933:ll_write_end()) page@000000001627eeab[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 89] 00000080:00200000:2.0:1787827204.833235:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000363b0f58 17ffffc0001001 3:0 ffff8ab3f042f3a0 89 lru 00000080:00200000:2.0:1787827204.833241:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000004f5d9efc 89: 1< 0 - - > 2< 364544 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833244:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000001627eeab 00000080:00200000:2.0:1787827204.833245:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 90. 00000080:00200000:2.0:1787827204.833247:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 90 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833261:0:48445:0:(rw26.c:875:ll_write_end()) pos 368640, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833266:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000e879c774[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 90] 00000080:00200000:2.0:1787827204.833269:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000c07445bb 17ffffc0001001 3:0 ffff8ab3f042f9f8 90 lru 00000080:00200000:2.0:1787827204.833275:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000713417f7 90: 1< 0 - - > 2< 368640 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833277:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000e879c774 00000080:00200000:2.0:1787827204.833279:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 91. 00000080:00200000:2.0:1787827204.833281:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 91 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833296:0:48445:0:(rw26.c:875:ll_write_end()) pos 372736, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833301:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000008aa76ab[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 91] 00000080:00200000:2.0:1787827204.833305:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000b0995fbd 17ffffc0001001 3:0 ffff8ab3f042f0e8 91 lru 00000080:00200000:2.0:1787827204.833311:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000947cb29a 91: 1< 0 - - > 2< 372736 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833313:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000008aa76ab 00000080:00200000:2.0:1787827204.833315:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 92. 00000080:00200000:2.0:1787827204.833317:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 92 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833330:0:48445:0:(rw26.c:875:ll_write_end()) pos 376832, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833334:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000ca9fa8a1[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 92] 00000080:00200000:2.0:1787827204.833338:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000008223f343 17ffffc0001001 3:0 ffff8ab3f042f740 92 lru 00000080:00200000:2.0:1787827204.833343:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000799e2296 92: 1< 0 - - > 2< 376832 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833346:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000ca9fa8a1 00000080:00200000:2.0:1787827204.833347:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 93. 00000080:00200000:2.0:1787827204.833349:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 93 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833372:0:48445:0:(rw26.c:875:ll_write_end()) pos 380928, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833378:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000069acc14e[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 93] 00000080:00200000:2.0:1787827204.833380:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000004ea052eb 17ffffc0001001 3:0 ffff8ab3d096a740 93 lru 00000080:00200000:2.0:1787827204.833385:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000081a1b0e6 93: 1< 0 - - > 2< 380928 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833387:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000069acc14e 00000080:00200000:2.0:1787827204.833389:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 94. 00000080:00200000:2.0:1787827204.833391:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 94 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833406:0:48445:0:(rw26.c:875:ll_write_end()) pos 385024, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833410:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000098b932c7[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 94] 00000080:00200000:2.0:1787827204.833413:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000004c4e2ef 17ffffc0001001 3:0 ffff8ab3d096a570 94 lru 00000080:00200000:2.0:1787827204.833419:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000003220307c 94: 1< 0 - - > 2< 385024 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833421:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000098b932c7 00000080:00200000:2.0:1787827204.833423:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 95. 00000080:00200000:2.0:1787827204.833425:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 95 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833441:0:48445:0:(rw26.c:875:ll_write_end()) pos 389120, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833446:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000075606c06[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 95] 00000080:00200000:2.0:1787827204.833449:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000e3e2e250 17ffffc0001001 3:0 ffff8ab3d096a1d0 95 lru 00000080:00200000:2.0:1787827204.833456:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000003202cc3 95: 1< 0 - - > 2< 389120 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833459:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000075606c06 00000080:00200000:2.0:1787827204.833460:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 96. 00000080:00200000:2.0:1787827204.833462:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 96 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833478:0:48445:0:(rw26.c:875:ll_write_end()) pos 393216, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833482:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000d8acf471[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 96] 00000080:00200000:2.0:1787827204.833485:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000001fbcd853 17ffffc0001001 3:0 ffff8ab3d096a658 96 lru 00000080:00200000:2.0:1787827204.833491:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000039ca7c04 96: 1< 0 - - > 2< 393216 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833493:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000d8acf471 00000080:00200000:2.0:1787827204.833495:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 97. 00000080:00200000:2.0:1787827204.833497:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 97 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833512:0:48445:0:(rw26.c:875:ll_write_end()) pos 397312, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833516:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000b78b0072[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 97] 00000080:00200000:2.0:1787827204.833519:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000009e6141c2 17ffffc0001001 3:0 ffff8ab3d096a000 97 lru 00000080:00200000:2.0:1787827204.833525:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000008eae4da8 97: 1< 0 - - > 2< 397312 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833528:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000b78b0072 00000080:00200000:2.0:1787827204.833529:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 98. 00000080:00200000:2.0:1787827204.833531:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 98 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833545:0:48445:0:(rw26.c:875:ll_write_end()) pos 401408, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833549:0:48445:0:(rw26.c:933:ll_write_end()) page@000000000df1f27a[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 98] 00000080:00200000:2.0:1787827204.833553:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000008eb9e407 17ffffc0001001 3:0 ffff8ab3d096ae80 98 lru 00000080:00200000:2.0:1787827204.833559:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000009fa0cf23 98: 1< 0 - - > 2< 401408 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833562:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000000df1f27a 00000080:00200000:2.0:1787827204.833563:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 99. 00000080:00200000:2.0:1787827204.833564:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 99 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833579:0:48445:0:(rw26.c:875:ll_write_end()) pos 405504, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833584:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000247ba5dd[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 99] 00000080:00200000:2.0:1787827204.833587:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000012ec3422 17ffffc0001021 2:0 ffff8ab3d096aae0 99 lru 00000080:00200000:2.0:1787827204.833593:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000ad89c742 99: 1< 0 - - > 2< 405504 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833596:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000247ba5dd 00000080:00200000:2.0:1787827204.833597:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 100. 00000080:00200000:2.0:1787827204.833599:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 100 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833611:0:48445:0:(rw26.c:875:ll_write_end()) pos 409600, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833615:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000016cc510[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 100] 00000080:00200000:2.0:1787827204.833618:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000008eddeeb7 17ffffc0001001 3:0 ffff8ab3d096abc8 100 lru 00000080:00200000:2.0:1787827204.833624:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000006c8f0c00 100: 1< 0 - - > 2< 409600 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833626:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000016cc510 00000080:00200000:2.0:1787827204.833627:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 101. 00000080:00200000:2.0:1787827204.833629:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 101 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833640:0:48445:0:(rw26.c:875:ll_write_end()) pos 413696, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833645:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000d7770a32[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 101] 00000080:00200000:2.0:1787827204.833648:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000bc09090a 17ffffc0001001 3:0 ffff8ab3d096a910 101 lru 00000080:00200000:2.0:1787827204.833654:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000003702d595 101: 1< 0 - - > 2< 413696 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833657:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000d7770a32 00000080:00200000:2.0:1787827204.833658:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 102. 00000080:00200000:2.0:1787827204.833660:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 102 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833670:0:48445:0:(rw26.c:875:ll_write_end()) pos 417792, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833675:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000065d42e4e[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 102] 00000080:00200000:2.0:1787827204.833678:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000003462c658 17ffffc0001001 3:0 ffff8ab3d096a828 102 lru 00000080:00200000:2.0:1787827204.833683:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000cfdd7dd9 102: 1< 0 - - > 2< 417792 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833685:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000065d42e4e 00000080:00200000:2.0:1787827204.833687:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 103. 00000080:00200000:2.0:1787827204.833689:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 103 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833702:0:48445:0:(rw26.c:875:ll_write_end()) pos 421888, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833705:0:48445:0:(rw26.c:933:ll_write_end()) page@000000001d0e589d[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 103] 00000080:00200000:2.0:1787827204.833707:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000f3b2f817 17ffffc0001001 3:0 ffff8ab3d096acb0 103 lru 00000080:00200000:2.0:1787827204.833711:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000f3778246 103: 1< 0 - - > 2< 421888 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833714:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000001d0e589d 00000080:00200000:2.0:1787827204.833715:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 104. 00000080:00200000:2.0:1787827204.833716:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 104 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833726:0:48445:0:(rw26.c:875:ll_write_end()) pos 425984, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833728:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000a1b0c83e[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 104] 00000080:00200000:2.0:1787827204.833731:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000001c31936 17ffffc0001001 3:0 ffff8ab3d096ad98 104 lru 00000080:00200000:2.0:1787827204.833735:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000003bc89755 104: 1< 0 - - > 2< 425984 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833736:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000a1b0c83e 00000080:00200000:2.0:1787827204.833737:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 105. 00000080:00200000:2.0:1787827204.833738:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 105 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833747:0:48445:0:(rw26.c:875:ll_write_end()) pos 430080, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833749:0:48445:0:(rw26.c:933:ll_write_end()) page@000000003ae34c25[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 105] 00000080:00200000:2.0:1787827204.833751:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000008222a873 17ffffc0001001 3:0 ffff8ab3d096a2b8 105 lru 00000080:00200000:2.0:1787827204.833754:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000f05cf231 105: 1< 0 - - > 2< 430080 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833756:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000003ae34c25 00000080:00200000:2.0:1787827204.833757:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 106. 00000080:00200000:2.0:1787827204.833758:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 106 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833767:0:48445:0:(rw26.c:875:ll_write_end()) pos 434176, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833770:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000df9fd8d0[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 106] 00000080:00200000:2.0:1787827204.833772:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000002194bc25 17ffffc0001001 3:0 ffff8ab3d096a488 106 lru 00000080:00200000:2.0:1787827204.833776:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000eeb837e5 106: 1< 0 - - > 2< 434176 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833777:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000df9fd8d0 00000080:00200000:2.0:1787827204.833778:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 107. 00000080:00200000:2.0:1787827204.833779:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 107 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833787:0:48445:0:(rw26.c:875:ll_write_end()) pos 438272, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833790:0:48445:0:(rw26.c:933:ll_write_end()) page@000000007cc9f035[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 107] 00000080:00200000:2.0:1787827204.833792:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000015a0f83a 17ffffc0001001 3:0 ffff8ab3d096a3a0 107 lru 00000080:00200000:2.0:1787827204.833795:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000e2dd70f9 107: 1< 0 - - > 2< 438272 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833796:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000007cc9f035 00000080:00200000:2.0:1787827204.833797:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 108. 00000080:00200000:2.0:1787827204.833798:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 108 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833808:0:48445:0:(rw26.c:875:ll_write_end()) pos 442368, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833811:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000024fe0e87[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 108] 00000080:00200000:2.0:1787827204.833813:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000043c2a2ee 17ffffc0001001 3:0 ffff8ab3d096a9f8 108 lru 00000080:00200000:2.0:1787827204.833819:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000004a7234d4 108: 1< 0 - - > 2< 442368 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833822:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000024fe0e87 00000080:00200000:2.0:1787827204.833823:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 109. 00000080:00200000:2.0:1787827204.833826:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 109 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833842:0:48445:0:(rw26.c:875:ll_write_end()) pos 446464, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833848:0:48445:0:(rw26.c:933:ll_write_end()) page@000000000356a1ae[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 109] 00000080:00200000:2.0:1787827204.833850:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000bf761815 17ffffc0001001 3:0 ffff8ab3d096a0e8 109 lru 00000080:00200000:2.0:1787827204.833855:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000003b731550 109: 1< 0 - - > 2< 446464 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833858:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000000356a1ae 00000080:00200000:2.0:1787827204.833859:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 110. 00000080:00200000:2.0:1787827204.833862:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 110 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833902:0:48445:0:(rw26.c:875:ll_write_end()) pos 450560, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833908:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000057ca8cd[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 110] 00000080:00200000:2.0:1787827204.833912:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000b85c2192 17ffffc0001001 3:0 ffff8ab3d0b3f1d0 110 lru 00000080:00200000:2.0:1787827204.833918:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000009b6b139f 110: 1< 0 - - > 2< 450560 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833921:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000057ca8cd 00000080:00200000:2.0:1787827204.833923:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 111. 00000080:00200000:2.0:1787827204.833925:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 111 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833942:0:48445:0:(rw26.c:875:ll_write_end()) pos 454656, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833947:0:48445:0:(rw26.c:933:ll_write_end()) page@000000001ab55cfd[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 111] 00000080:00200000:2.0:1787827204.833950:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000056fe8d95 17ffffc0001001 3:0 ffff8ab3d0b3f658 111 lru 00000080:00200000:2.0:1787827204.833954:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000e8f3b6ad 111: 1< 0 - - > 2< 454656 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833956:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000001ab55cfd 00000080:00200000:2.0:1787827204.833957:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 112. 00000080:00200000:2.0:1787827204.833959:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 112 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833969:0:48445:0:(rw26.c:875:ll_write_end()) pos 458752, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833972:0:48445:0:(rw26.c:933:ll_write_end()) page@000000008b15098b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 112] 00000080:00200000:2.0:1787827204.833974:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000a235a91c 17ffffc0001001 3:0 ffff8ab3d0b3f000 112 lru 00000080:00200000:2.0:1787827204.833977:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000062e1197e 112: 1< 0 - - > 2< 458752 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.833980:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000008b15098b 00000080:00200000:2.0:1787827204.833981:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 113. 00000080:00200000:2.0:1787827204.833982:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 113 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.833990:0:48445:0:(rw26.c:875:ll_write_end()) pos 462848, len 4096, copied 4096 00000080:00200000:2.0:1787827204.833993:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000dec42858[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 113] 00000080:00200000:2.0:1787827204.833995:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000017b24084 17ffffc0001001 3:0 ffff8ab3d0b3fe80 113 lru 00000080:00200000:2.0:1787827204.833999:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000567d1841 113: 1< 0 - - > 2< 462848 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834000:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000dec42858 00000080:00200000:2.0:1787827204.834001:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 114. 00000080:00200000:2.0:1787827204.834002:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 114 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834013:0:48445:0:(rw26.c:875:ll_write_end()) pos 466944, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834016:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000053fffdbc[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 114] 00000080:00200000:2.0:1787827204.834018:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000004d0c9348 17ffffc0001021 2:0 ffff8ab3d0b3fae0 114 lru 00000080:00200000:2.0:1787827204.834021:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000085d4cb5e 114: 1< 0 - - > 2< 466944 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834023:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000053fffdbc 00000080:00200000:2.0:1787827204.834024:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 115. 00000080:00200000:2.0:1787827204.834025:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 115 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834034:0:48445:0:(rw26.c:875:ll_write_end()) pos 471040, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834036:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000cf133d3c[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 115] 00000080:00200000:2.0:1787827204.834038:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000915de6ca 17ffffc0001001 3:0 ffff8ab3d0b3fbc8 115 lru 00000080:00200000:2.0:1787827204.834042:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000008b4209 115: 1< 0 - - > 2< 471040 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834043:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000cf133d3c 00000080:00200000:2.0:1787827204.834044:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 116. 00000080:00200000:2.0:1787827204.834045:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 116 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834055:0:48445:0:(rw26.c:875:ll_write_end()) pos 475136, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834058:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000072764cbd[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 116] 00000080:00200000:2.0:1787827204.834060:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000d1cace5e 17ffffc0001001 3:0 ffff8ab3d0b3f910 116 lru 00000080:00200000:2.0:1787827204.834063:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000049f702c5 116: 1< 0 - - > 2< 475136 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834065:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000072764cbd 00000080:00200000:2.0:1787827204.834066:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 117. 00000080:00200000:2.0:1787827204.834067:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 117 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834078:0:48445:0:(rw26.c:875:ll_write_end()) pos 479232, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834080:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000da215ead[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 117] 00000080:00200000:2.0:1787827204.834082:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000bb9b25da 17ffffc0001001 3:0 ffff8ab3d0b3f828 117 lru 00000080:00200000:2.0:1787827204.834086:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000dce74c45 117: 1< 0 - - > 2< 479232 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834088:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000da215ead 00000080:00200000:2.0:1787827204.834088:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 118. 00000080:00200000:2.0:1787827204.834089:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 118 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834113:0:48445:0:(rw26.c:875:ll_write_end()) pos 483328, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834117:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000021438302[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 118] 00000080:00200000:2.0:1787827204.834119:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000502494c0 17ffffc0001001 3:0 ffff8ab3d0b3fcb0 118 lru 00000080:00200000:2.0:1787827204.834122:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000be4ecf33 118: 1< 0 - - > 2< 483328 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834124:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000021438302 00000080:00200000:2.0:1787827204.834125:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 119. 00000080:00200000:2.0:1787827204.834126:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 119 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834136:0:48445:0:(rw26.c:875:ll_write_end()) pos 487424, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834138:0:48445:0:(rw26.c:933:ll_write_end()) page@000000003fe2c3a5[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 119] 00000080:00200000:2.0:1787827204.834140:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000e1747fbf 17ffffc0001001 3:0 ffff8ab3d0b3fd98 119 lru 00000080:00200000:2.0:1787827204.834144:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000000bf037e4 119: 1< 0 - - > 2< 487424 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834146:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000003fe2c3a5 00000080:00200000:2.0:1787827204.834147:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 120. 00000080:00200000:2.0:1787827204.834148:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 120 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834156:0:48445:0:(rw26.c:875:ll_write_end()) pos 491520, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834159:0:48445:0:(rw26.c:933:ll_write_end()) page@000000006ccd2fd2[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 120] 00000080:00200000:2.0:1787827204.834160:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000091da5ee8 17ffffc0001001 3:0 ffff8ab3d0b3f2b8 120 lru 00000080:00200000:2.0:1787827204.834164:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000005d4f290e 120: 1< 0 - - > 2< 491520 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834166:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000006ccd2fd2 00000080:00200000:2.0:1787827204.834167:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 121. 00000080:00200000:2.0:1787827204.834168:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 121 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834176:0:48445:0:(rw26.c:875:ll_write_end()) pos 495616, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834179:0:48445:0:(rw26.c:933:ll_write_end()) page@000000008a09e176[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 121] 00000080:00200000:2.0:1787827204.834181:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000001a7e64f1 17ffffc0001001 3:0 ffff8ab3d0b3f488 121 lru 00000080:00200000:2.0:1787827204.834185:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000ecb4ced2 121: 1< 0 - - > 2< 495616 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834186:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000008a09e176 00000080:00200000:2.0:1787827204.834187:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 122. 00000080:00200000:2.0:1787827204.834188:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 122 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834197:0:48445:0:(rw26.c:875:ll_write_end()) pos 499712, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834199:0:48445:0:(rw26.c:933:ll_write_end()) page@000000000a79643c[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 122] 00000080:00200000:2.0:1787827204.834201:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000d3db3ae3 17ffffc0001001 3:0 ffff8ab3d0b3f3a0 122 lru 00000080:00200000:2.0:1787827204.834205:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000000c85dd65 122: 1< 0 - - > 2< 499712 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834206:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000000a79643c 00000080:00200000:2.0:1787827204.834206:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 123. 00000080:00200000:2.0:1787827204.834207:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 123 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834215:0:48445:0:(rw26.c:875:ll_write_end()) pos 503808, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834218:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000096971200[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 123] 00000080:00200000:2.0:1787827204.834220:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000034c3e1a2 17ffffc0001001 3:0 ffff8ab3d0b3f9f8 123 lru 00000080:00200000:2.0:1787827204.834223:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000986cd087 123: 1< 0 - - > 2< 503808 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834225:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000096971200 00000080:00200000:2.0:1787827204.834226:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 124. 00000080:00200000:2.0:1787827204.834227:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 124 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834237:0:48445:0:(rw26.c:875:ll_write_end()) pos 507904, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834239:0:48445:0:(rw26.c:933:ll_write_end()) page@000000009f90afb2[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 124] 00000080:00200000:2.0:1787827204.834241:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000053e33982 17ffffc0001001 3:0 ffff8ab3d0b3f0e8 124 lru 00000080:00200000:2.0:1787827204.834245:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000002b2135b 124: 1< 0 - - > 2< 507904 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834246:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000009f90afb2 00000080:00200000:2.0:1787827204.834247:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 125. 00000080:00200000:2.0:1787827204.834248:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 125 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834256:0:48445:0:(rw26.c:875:ll_write_end()) pos 512000, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834259:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000fdd946e2[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 125] 00000080:00200000:2.0:1787827204.834261:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000904dd6bc 17ffffc0001001 3:0 ffff8ab3d0b3f740 125 lru 00000080:00200000:2.0:1787827204.834265:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000c07c696a 125: 1< 0 - - > 2< 512000 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834267:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000fdd946e2 00000080:00200000:2.0:1787827204.834268:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 126. 00000080:00200000:2.0:1787827204.834269:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 126 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834278:0:48445:0:(rw26.c:875:ll_write_end()) pos 516096, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834281:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000f5b065dc[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 126] 00000080:00200000:2.0:1787827204.834283:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000a55a219f 17ffffc0001001 3:0 ffff8ab3d0b3f570 126 lru 00000080:00200000:2.0:1787827204.834287:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000ee1f62a0 126: 1< 0 - - > 2< 516096 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834288:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000f5b065dc 00000080:00200000:2.0:1787827204.834289:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 127. 00000080:00200000:2.0:1787827204.834290:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 127 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834304:0:48445:0:(rw26.c:875:ll_write_end()) pos 520192, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834306:0:48445:0:(rw26.c:933:ll_write_end()) page@000000008115a21b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 127] 00000080:00200000:2.0:1787827204.834308:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000002b554cf0 17ffffc0001001 3:0 ffff8ab3c69d8ae0 127 lru 00000080:00200000:2.0:1787827204.834312:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000f71f208e 127: 1< 0 - - > 2< 520192 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834313:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000008115a21b 00000080:00200000:2.0:1787827204.834314:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 128. 00000080:00200000:2.0:1787827204.834315:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 128 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834325:0:48445:0:(rw26.c:875:ll_write_end()) pos 524288, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834328:0:48445:0:(rw26.c:933:ll_write_end()) page@000000003da79c7b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 128] 00000080:00200000:2.0:1787827204.834330:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000024c25548 17ffffc0001001 3:0 ffff8ab3c69d8bc8 128 lru 00000080:00200000:2.0:1787827204.834334:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000f0e67310 128: 1< 0 - - > 2< 524288 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834336:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000003da79c7b 00000080:00200000:2.0:1787827204.834336:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 129. 00000080:00200000:2.0:1787827204.834337:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 129 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834348:0:48445:0:(rw26.c:875:ll_write_end()) pos 528384, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834351:0:48445:0:(rw26.c:933:ll_write_end()) page@000000005e876271[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 129] 00000080:00200000:2.0:1787827204.834353:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000fa371044 17ffffc0001021 2:0 ffff8ab3c69d8910 129 lru 00000080:00200000:2.0:1787827204.834356:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000d7be7ae0 129: 1< 0 - - > 2< 528384 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834358:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000005e876271 00000080:00200000:2.0:1787827204.834358:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 130. 00000080:00200000:2.0:1787827204.834360:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 130 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834368:0:48445:0:(rw26.c:875:ll_write_end()) pos 532480, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834371:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000d9b171db[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 130] 00000080:00200000:2.0:1787827204.834373:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000002bc33f04 17ffffc0001001 3:0 ffff8ab3c69d8828 130 lru 00000080:00200000:2.0:1787827204.834377:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000067032774 130: 1< 0 - - > 2< 532480 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834379:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000d9b171db 00000080:00200000:2.0:1787827204.834380:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 131. 00000080:00200000:2.0:1787827204.834381:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 131 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834391:0:48445:0:(rw26.c:875:ll_write_end()) pos 536576, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834394:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000038710a99[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 131] 00000080:00200000:2.0:1787827204.834396:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000982e1006 17ffffc0001001 3:0 ffff8ab3c69d8cb0 131 lru 00000080:00200000:2.0:1787827204.834400:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000081593a14 131: 1< 0 - - > 2< 536576 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834401:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000038710a99 00000080:00200000:2.0:1787827204.834402:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 132. 00000080:00200000:2.0:1787827204.834403:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 132 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834412:0:48445:0:(rw26.c:875:ll_write_end()) pos 540672, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834414:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000243dad4b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 132] 00000080:00200000:2.0:1787827204.834416:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000002cc4761a 17ffffc0001001 3:0 ffff8ab3c69d8d98 132 lru 00000080:00200000:2.0:1787827204.834419:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000ac7de63a 132: 1< 0 - - > 2< 540672 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834421:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000243dad4b 00000080:00200000:2.0:1787827204.834422:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 133. 00000080:00200000:2.0:1787827204.834423:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 133 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834431:0:48445:0:(rw26.c:875:ll_write_end()) pos 544768, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834434:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000e66641e3[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 133] 00000080:00200000:2.0:1787827204.834436:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000065d9762d 17ffffc0001001 3:0 ffff8ab3c69d82b8 133 lru 00000080:00200000:2.0:1787827204.834440:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000258d8f97 133: 1< 0 - - > 2< 544768 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834442:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000e66641e3 00000080:00200000:2.0:1787827204.834443:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 134. 00000080:00200000:2.0:1787827204.834444:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 134 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834452:0:48445:0:(rw26.c:875:ll_write_end()) pos 548864, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834454:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000008143a7c[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 134] 00000080:00200000:2.0:1787827204.834456:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000006555c29a 17ffffc0001001 3:0 ffff8ab3c69d8488 134 lru 00000080:00200000:2.0:1787827204.834460:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000d5e8106c 134: 1< 0 - - > 2< 548864 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834461:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000008143a7c 00000080:00200000:2.0:1787827204.834462:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 135. 00000080:00200000:2.0:1787827204.834463:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 135 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834473:0:48445:0:(rw26.c:875:ll_write_end()) pos 552960, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834477:0:48445:0:(rw26.c:933:ll_write_end()) page@000000006210042c[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 135] 00000080:00200000:2.0:1787827204.834481:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000021d52e0a 17ffffc0001001 3:0 ffff8ab3c69d83a0 135 lru 00000080:00200000:2.0:1787827204.834487:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000fd440abd 135: 1< 0 - - > 2< 552960 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834490:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000006210042c 00000080:00200000:2.0:1787827204.834491:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 136. 00000080:00200000:2.0:1787827204.834494:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 136 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834508:0:48445:0:(rw26.c:875:ll_write_end()) pos 557056, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834513:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000050cc0e12[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 136] 00000080:00200000:2.0:1787827204.834516:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000f4b3b649 17ffffc0001001 3:0 ffff8ab3c69d89f8 136 lru 00000080:00200000:2.0:1787827204.834523:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000002fc686a1 136: 1< 0 - - > 2< 557056 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834525:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000050cc0e12 00000080:00200000:2.0:1787827204.834527:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 137. 00000080:00200000:2.0:1787827204.834529:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 137 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834542:0:48445:0:(rw26.c:875:ll_write_end()) pos 561152, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834547:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000a6ed269b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 137] 00000080:00200000:2.0:1787827204.834550:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000001de11dc2 17ffffc0001001 3:0 ffff8ab3c69d80e8 137 lru 00000080:00200000:2.0:1787827204.834556:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000be1ae63e 137: 1< 0 - - > 2< 561152 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834559:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000a6ed269b 00000080:00200000:2.0:1787827204.834560:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 138. 00000080:00200000:2.0:1787827204.834562:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 138 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834575:0:48445:0:(rw26.c:875:ll_write_end()) pos 565248, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834579:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000c5c74bfe[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 138] 00000080:00200000:2.0:1787827204.834582:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000093c4723b 17ffffc0001001 3:0 ffff8ab3c69d8740 138 lru 00000080:00200000:2.0:1787827204.834588:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000003d5688dc 138: 1< 0 - - > 2< 565248 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834591:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000c5c74bfe 00000080:00200000:2.0:1787827204.834592:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 139. 00000080:00200000:2.0:1787827204.834594:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 139 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834606:0:48445:0:(rw26.c:875:ll_write_end()) pos 569344, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834609:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000000c59b9b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 139] 00000080:00200000:2.0:1787827204.834612:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000c8b4800b 17ffffc0001001 3:0 ffff8ab3c69d8570 139 lru 00000080:00200000:2.0:1787827204.834615:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000527ee24c 139: 1< 0 - - > 2< 569344 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834617:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000000c59b9b 00000080:00200000:2.0:1787827204.834618:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 140. 00000080:00200000:2.0:1787827204.834620:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 140 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834629:0:48445:0:(rw26.c:875:ll_write_end()) pos 573440, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834632:0:48445:0:(rw26.c:933:ll_write_end()) page@000000007c5c39ce[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 140] 00000080:00200000:2.0:1787827204.834634:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000a94992ff 17ffffc0001001 3:0 ffff8ab3c69d81d0 140 lru 00000080:00200000:2.0:1787827204.834638:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000be5a479f 140: 1< 0 - - > 2< 573440 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834639:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000007c5c39ce 00000080:00200000:2.0:1787827204.834640:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 141. 00000080:00200000:2.0:1787827204.834641:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 141 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834650:0:48445:0:(rw26.c:875:ll_write_end()) pos 577536, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834652:0:48445:0:(rw26.c:933:ll_write_end()) page@000000003ccd00a5[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 141] 00000080:00200000:2.0:1787827204.834654:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000001a13c543 17ffffc0001001 3:0 ffff8ab3c69d8658 141 lru 00000080:00200000:2.0:1787827204.834657:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000032abed6c 141: 1< 0 - - > 2< 577536 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834659:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000003ccd00a5 00000080:00200000:2.0:1787827204.834660:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 142. 00000080:00200000:2.0:1787827204.834661:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 142 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834670:0:48445:0:(rw26.c:875:ll_write_end()) pos 581632, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834672:0:48445:0:(rw26.c:933:ll_write_end()) page@000000004e765585[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 142] 00000080:00200000:2.0:1787827204.834674:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000000b412b33 17ffffc0001001 3:0 ffff8ab3c69d8000 142 lru 00000080:00200000:2.0:1787827204.834679:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000001755d1e7 142: 1< 0 - - > 2< 581632 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834680:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000004e765585 00000080:00200000:2.0:1787827204.834681:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 143. 00000080:00200000:2.0:1787827204.834682:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 143 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834690:0:48445:0:(rw26.c:875:ll_write_end()) pos 585728, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834693:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000070a4a259[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 143] 00000080:00200000:2.0:1787827204.834694:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000b7cc4eaa 17ffffc0001001 3:0 ffff8ab3c69d8e80 143 lru 00000080:00200000:2.0:1787827204.834698:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000b0289ce2 143: 1< 0 - - > 2< 585728 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834699:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000070a4a259 00000080:00200000:2.0:1787827204.834700:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 144. 00000080:00200000:2.0:1787827204.834702:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 144 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834719:0:48445:0:(rw26.c:875:ll_write_end()) pos 589824, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834722:0:48445:0:(rw26.c:933:ll_write_end()) page@000000000e05c12a[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 144] 00000080:00200000:2.0:1787827204.834724:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000a608c89d 17ffffc0001021 2:0 ffff8ab3d2346d98 144 lru 00000080:00200000:2.0:1787827204.834728:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000000b6816d9 144: 1< 0 - - > 2< 589824 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834729:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000000e05c12a 00000080:00200000:2.0:1787827204.834730:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 145. 00000080:00200000:2.0:1787827204.834731:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 145 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834742:0:48445:0:(rw26.c:875:ll_write_end()) pos 593920, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834744:0:48445:0:(rw26.c:933:ll_write_end()) page@000000006afebffd[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 145] 00000080:00200000:2.0:1787827204.834746:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000001946b7a3 17ffffc0001001 3:0 ffff8ab3d23462b8 145 lru 00000080:00200000:2.0:1787827204.834749:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000489801cb 145: 1< 0 - - > 2< 593920 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834751:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000006afebffd 00000080:00200000:2.0:1787827204.834752:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 146. 00000080:00200000:2.0:1787827204.834753:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 146 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834762:0:48445:0:(rw26.c:875:ll_write_end()) pos 598016, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834765:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000010f633b2[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 146] 00000080:00200000:2.0:1787827204.834767:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000c98472f8 17ffffc0001001 3:0 ffff8ab3d2346488 146 lru 00000080:00200000:2.0:1787827204.834770:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000001f5c1245 146: 1< 0 - - > 2< 598016 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834772:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000010f633b2 00000080:00200000:2.0:1787827204.834773:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 147. 00000080:00200000:2.0:1787827204.834774:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 147 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834783:0:48445:0:(rw26.c:875:ll_write_end()) pos 602112, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834785:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000489bdafd[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 147] 00000080:00200000:2.0:1787827204.834787:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000fc92439c 17ffffc0001001 3:0 ffff8ab3d23463a0 147 lru 00000080:00200000:2.0:1787827204.834790:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000001a2ff6fa 147: 1< 0 - - > 2< 602112 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834792:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000489bdafd 00000080:00200000:2.0:1787827204.834792:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 148. 00000080:00200000:2.0:1787827204.834794:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 148 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834802:0:48445:0:(rw26.c:875:ll_write_end()) pos 606208, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834805:0:48445:0:(rw26.c:933:ll_write_end()) page@000000002c681c8d[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 148] 00000080:00200000:2.0:1787827204.834807:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000097bd354a 17ffffc0001001 3:0 ffff8ab3d23469f8 148 lru 00000080:00200000:2.0:1787827204.834811:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000c16b32c9 148: 1< 0 - - > 2< 606208 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834813:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000002c681c8d 00000080:00200000:2.0:1787827204.834814:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 149. 00000080:00200000:2.0:1787827204.834815:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 149 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834824:0:48445:0:(rw26.c:875:ll_write_end()) pos 610304, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834827:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000e01f9906[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 149] 00000080:00200000:2.0:1787827204.834829:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000ba660656 17ffffc0001001 3:0 ffff8ab3d23460e8 149 lru 00000080:00200000:2.0:1787827204.834833:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000006dacfd76 149: 1< 0 - - > 2< 610304 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834835:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000e01f9906 00000080:00200000:2.0:1787827204.834835:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 150. 00000080:00200000:2.0:1787827204.834836:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 150 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834845:0:48445:0:(rw26.c:875:ll_write_end()) pos 614400, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834847:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000e15ad5b6[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 150] 00000080:00200000:2.0:1787827204.834849:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000020b819bf 17ffffc0001001 3:0 ffff8ab3d2346740 150 lru 00000080:00200000:2.0:1787827204.834852:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000d7a56687 150: 1< 0 - - > 2< 614400 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834854:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000e15ad5b6 00000080:00200000:2.0:1787827204.834855:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 151. 00000080:00200000:2.0:1787827204.834856:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 151 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834864:0:48445:0:(rw26.c:875:ll_write_end()) pos 618496, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834867:0:48445:0:(rw26.c:933:ll_write_end()) page@000000004f760b84[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 151] 00000080:00200000:2.0:1787827204.834869:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000097810cf9 17ffffc0001001 3:0 ffff8ab3d2346570 151 lru 00000080:00200000:2.0:1787827204.834873:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000468a7caa 151: 1< 0 - - > 2< 618496 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834875:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000004f760b84 00000080:00200000:2.0:1787827204.834876:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 152. 00000080:00200000:2.0:1787827204.834877:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 152 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834902:0:48445:0:(rw26.c:875:ll_write_end()) pos 622592, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834906:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000bb7ff49b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 152] 00000080:00200000:2.0:1787827204.834907:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000008c2b4f01 17ffffc0001001 3:0 ffff8ab3d23461d0 152 lru 00000080:00200000:2.0:1787827204.834911:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000c86e83c1 152: 1< 0 - - > 2< 622592 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834913:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000bb7ff49b 00000080:00200000:2.0:1787827204.834914:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 153. 00000080:00200000:2.0:1787827204.834915:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 153 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834926:0:48445:0:(rw26.c:875:ll_write_end()) pos 626688, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834929:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000076c2ca75[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 153] 00000080:00200000:2.0:1787827204.834931:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000e7044cd3 17ffffc0001001 3:0 ffff8ab3d2346658 153 lru 00000080:00200000:2.0:1787827204.834935:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000b0c53adb 153: 1< 0 - - > 2< 626688 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834936:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000076c2ca75 00000080:00200000:2.0:1787827204.834937:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 154. 00000080:00200000:2.0:1787827204.834938:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 154 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834947:0:48445:0:(rw26.c:875:ll_write_end()) pos 630784, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834950:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000111417c2[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 154] 00000080:00200000:2.0:1787827204.834951:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000dbfe8644 17ffffc0001001 3:0 ffff8ab3d2346000 154 lru 00000080:00200000:2.0:1787827204.834955:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000b7038190 154: 1< 0 - - > 2< 630784 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834956:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000111417c2 00000080:00200000:2.0:1787827204.834957:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 155. 00000080:00200000:2.0:1787827204.834958:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 155 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834967:0:48445:0:(rw26.c:875:ll_write_end()) pos 634880, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834969:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000018447ed0[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 155] 00000080:00200000:2.0:1787827204.834971:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000000422d665 17ffffc0001001 3:0 ffff8ab3d2346e80 155 lru 00000080:00200000:2.0:1787827204.834975:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000032e45915 155: 1< 0 - - > 2< 634880 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834977:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000018447ed0 00000080:00200000:2.0:1787827204.834977:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 156. 00000080:00200000:2.0:1787827204.834978:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 156 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.834986:0:48445:0:(rw26.c:875:ll_write_end()) pos 638976, len 4096, copied 4096 00000080:00200000:2.0:1787827204.834989:0:48445:0:(rw26.c:933:ll_write_end()) page@000000009d03eefe[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 156] 00000080:00200000:2.0:1787827204.834991:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000060aedec4 17ffffc0001001 3:0 ffff8ab3d2346ae0 156 lru 00000080:00200000:2.0:1787827204.834994:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000001a2ec85 156: 1< 0 - - > 2< 638976 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.834996:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000009d03eefe 00000080:00200000:2.0:1787827204.834996:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 157. 00000080:00200000:2.0:1787827204.834997:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 157 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835006:0:48445:0:(rw26.c:875:ll_write_end()) pos 643072, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835009:0:48445:0:(rw26.c:933:ll_write_end()) page@000000000913de2c[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 157] 00000080:00200000:2.0:1787827204.835011:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000081fa0a03 17ffffc0001001 3:0 ffff8ab3d2346bc8 157 lru 00000080:00200000:2.0:1787827204.835014:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000005a1193bc 157: 1< 0 - - > 2< 643072 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835016:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000000913de2c 00000080:00200000:2.0:1787827204.835017:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 158. 00000080:00200000:2.0:1787827204.835018:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 158 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835027:0:48445:0:(rw26.c:875:ll_write_end()) pos 647168, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835029:0:48445:0:(rw26.c:933:ll_write_end()) page@000000001c340325[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 158] 00000080:00200000:2.0:1787827204.835031:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000073af1e6f 17ffffc0001001 3:0 ffff8ab3d2346910 158 lru 00000080:00200000:2.0:1787827204.835035:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000da4416fc 158: 1< 0 - - > 2< 647168 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835037:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000001c340325 00000080:00200000:2.0:1787827204.835038:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 159. 00000080:00200000:2.0:1787827204.835039:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 159 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835051:0:48445:0:(rw26.c:875:ll_write_end()) pos 651264, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835054:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000e6419e1d[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 159] 00000080:00200000:2.0:1787827204.835055:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000005eab26b7 17ffffc0001021 2:0 ffff8ab3d2346828 159 lru 00000080:00200000:2.0:1787827204.835059:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000c8b223d6 159: 1< 0 - - > 2< 651264 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835060:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000e6419e1d 00000080:00200000:2.0:1787827204.835061:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 160. 00000080:00200000:2.0:1787827204.835063:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 160 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835071:0:48445:0:(rw26.c:875:ll_write_end()) pos 655360, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835074:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000910e483b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 160] 00000080:00200000:2.0:1787827204.835075:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000b6ea3ae5 17ffffc0001001 3:0 ffff8ab3d2346cb0 160 lru 00000080:00200000:2.0:1787827204.835079:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000ba39efaf 160: 1< 0 - - > 2< 655360 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835081:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000910e483b 00000080:00200000:2.0:1787827204.835082:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 161. 00000080:00200000:2.0:1787827204.835083:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 161 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835109:0:48445:0:(rw26.c:875:ll_write_end()) pos 659456, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835113:0:48445:0:(rw26.c:933:ll_write_end()) page@000000002ec5d044[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 161] 00000080:00200000:2.0:1787827204.835115:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000f40078fb 17ffffc0001001 3:0 ffff8ab3c82a5740 161 lru 00000080:00200000:2.0:1787827204.835118:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000f8840214 161: 1< 0 - - > 2< 659456 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835120:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000002ec5d044 00000080:00200000:2.0:1787827204.835121:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 162. 00000080:00200000:2.0:1787827204.835122:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 162 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835133:0:48445:0:(rw26.c:875:ll_write_end()) pos 663552, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835136:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000bbea54be[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 162] 00000080:00200000:2.0:1787827204.835138:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000074a84b6 17ffffc0001001 3:0 ffff8ab3c82a5570 162 lru 00000080:00200000:2.0:1787827204.835142:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000005bde4ee 162: 1< 0 - - > 2< 663552 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835144:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000bbea54be 00000080:00200000:2.0:1787827204.835144:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 163. 00000080:00200000:2.0:1787827204.835145:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 163 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835154:0:48445:0:(rw26.c:875:ll_write_end()) pos 667648, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835156:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000cb15c60e[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 163] 00000080:00200000:2.0:1787827204.835158:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000064573f5d 17ffffc0001001 3:0 ffff8ab3c82a51d0 163 lru 00000080:00200000:2.0:1787827204.835161:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000008e3a3c80 163: 1< 0 - - > 2< 667648 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835163:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000cb15c60e 00000080:00200000:2.0:1787827204.835163:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 164. 00000080:00200000:2.0:1787827204.835165:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 164 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835173:0:48445:0:(rw26.c:875:ll_write_end()) pos 671744, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835176:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000762daeca[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 164] 00000080:00200000:2.0:1787827204.835178:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000110583ef 17ffffc0001001 3:0 ffff8ab3c82a5658 164 lru 00000080:00200000:2.0:1787827204.835181:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000069a8de71 164: 1< 0 - - > 2< 671744 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835183:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000762daeca 00000080:00200000:2.0:1787827204.835183:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 165. 00000080:00200000:2.0:1787827204.835185:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 165 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835193:0:48445:0:(rw26.c:875:ll_write_end()) pos 675840, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835195:0:48445:0:(rw26.c:933:ll_write_end()) page@000000008e985816[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 165] 00000080:00200000:2.0:1787827204.835197:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000520e707b 17ffffc0001001 3:0 ffff8ab3c82a5000 165 lru 00000080:00200000:2.0:1787827204.835201:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000004281c67e 165: 1< 0 - - > 2< 675840 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835202:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000008e985816 00000080:00200000:2.0:1787827204.835203:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 166. 00000080:00200000:2.0:1787827204.835204:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 166 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835214:0:48445:0:(rw26.c:875:ll_write_end()) pos 679936, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835217:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000662f6584[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 166] 00000080:00200000:2.0:1787827204.835219:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000071c0d480 17ffffc0001001 3:0 ffff8ab3c82a5e80 166 lru 00000080:00200000:2.0:1787827204.835223:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000042524cc6 166: 1< 0 - - > 2< 679936 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835225:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000662f6584 00000080:00200000:2.0:1787827204.835226:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 167. 00000080:00200000:2.0:1787827204.835227:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 167 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835236:0:48445:0:(rw26.c:875:ll_write_end()) pos 684032, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835238:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000ce3226e5[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 167] 00000080:00200000:2.0:1787827204.835241:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000d75f7db4 17ffffc0001001 3:0 ffff8ab3c82a5ae0 167 lru 00000080:00200000:2.0:1787827204.835244:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000007bd27ad 167: 1< 0 - - > 2< 684032 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835246:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000ce3226e5 00000080:00200000:2.0:1787827204.835247:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 168. 00000080:00200000:2.0:1787827204.835248:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 168 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835256:0:48445:0:(rw26.c:875:ll_write_end()) pos 688128, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835259:0:48445:0:(rw26.c:933:ll_write_end()) page@000000004311d3c8[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 168] 00000080:00200000:2.0:1787827204.835261:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000009bad148f 17ffffc0001001 3:0 ffff8ab3c82a5bc8 168 lru 00000080:00200000:2.0:1787827204.835264:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000078e8bd2a 168: 1< 0 - - > 2< 688128 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835266:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000004311d3c8 00000080:00200000:2.0:1787827204.835266:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 169. 00000080:00200000:2.0:1787827204.835268:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 169 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835276:0:48445:0:(rw26.c:875:ll_write_end()) pos 692224, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835279:0:48445:0:(rw26.c:933:ll_write_end()) page@000000005a484ccf[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 169] 00000080:00200000:2.0:1787827204.835281:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000daa69c00 17ffffc0001001 3:0 ffff8ab3c82a5910 169 lru 00000080:00200000:2.0:1787827204.835285:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000df2c2ce4 169: 1< 0 - - > 2< 692224 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835287:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000005a484ccf 00000080:00200000:2.0:1787827204.835288:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 170. 00000080:00200000:2.0:1787827204.835289:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 170 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835297:0:48445:0:(rw26.c:875:ll_write_end()) pos 696320, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835299:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000e717d759[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 170] 00000080:00200000:2.0:1787827204.835301:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000005133df81 17ffffc0001001 3:0 ffff8ab3c82a5828 170 lru 00000080:00200000:2.0:1787827204.835305:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000f2a0c461 170: 1< 0 - - > 2< 696320 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835306:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000e717d759 00000080:00200000:2.0:1787827204.835307:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 171. 00000080:00200000:2.0:1787827204.835308:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 171 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835317:0:48445:0:(rw26.c:875:ll_write_end()) pos 700416, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835320:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000f94e9d01[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 171] 00000080:00200000:2.0:1787827204.835322:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000000c5aae93 17ffffc0001001 3:0 ffff8ab3c82a5cb0 171 lru 00000080:00200000:2.0:1787827204.835326:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000006c39fd6c 171: 1< 0 - - > 2< 700416 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835327:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000f94e9d01 00000080:00200000:2.0:1787827204.835328:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 172. 00000080:00200000:2.0:1787827204.835329:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 172 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835337:0:48445:0:(rw26.c:875:ll_write_end()) pos 704512, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835340:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000b89921b8[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 172] 00000080:00200000:2.0:1787827204.835342:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000002d470adf 17ffffc0001001 3:0 ffff8ab3c82a5d98 172 lru 00000080:00200000:2.0:1787827204.835345:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000000dffa2d0 172: 1< 0 - - > 2< 704512 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835346:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000b89921b8 00000080:00200000:2.0:1787827204.835347:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 173. 00000080:00200000:2.0:1787827204.835349:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 173 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835358:0:48445:0:(rw26.c:875:ll_write_end()) pos 708608, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835361:0:48445:0:(rw26.c:933:ll_write_end()) page@000000002e1f836f[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 173] 00000080:00200000:2.0:1787827204.835363:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000c8863fbd 17ffffc0001001 3:0 ffff8ab3c82a52b8 173 lru 00000080:00200000:2.0:1787827204.835367:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000009964aa41 173: 1< 0 - - > 2< 708608 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835369:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000002e1f836f 00000080:00200000:2.0:1787827204.835369:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 174. 00000080:00200000:2.0:1787827204.835370:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 174 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835381:0:48445:0:(rw26.c:875:ll_write_end()) pos 712704, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835383:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000042bf7797[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 174] 00000080:00200000:2.0:1787827204.835385:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000006ea468e1 17ffffc0001021 2:0 ffff8ab3c82a5488 174 lru 00000080:00200000:2.0:1787827204.835388:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000004638b757 174: 1< 0 - - > 2< 712704 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835390:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000042bf7797 00000080:00200000:2.0:1787827204.835391:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 175. 00000080:00200000:2.0:1787827204.835392:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 175 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835400:0:48445:0:(rw26.c:875:ll_write_end()) pos 716800, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835403:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000023aedbbd[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 175] 00000080:00200000:2.0:1787827204.835405:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000ab8106e5 17ffffc0001001 3:0 ffff8ab3c82a53a0 175 lru 00000080:00200000:2.0:1787827204.835409:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000d0523851 175: 1< 0 - - > 2< 716800 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835411:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000023aedbbd 00000080:00200000:2.0:1787827204.835412:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 176. 00000080:00200000:2.0:1787827204.835413:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 176 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835421:0:48445:0:(rw26.c:875:ll_write_end()) pos 720896, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835424:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000dcd5a39a[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 176] 00000080:00200000:2.0:1787827204.835426:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000b0ea87a7 17ffffc0001001 3:0 ffff8ab3c82a59f8 176 lru 00000080:00200000:2.0:1787827204.835429:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000063a9a567 176: 1< 0 - - > 2< 720896 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835430:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000dcd5a39a 00000080:00200000:2.0:1787827204.835431:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 177. 00000080:00200000:2.0:1787827204.835432:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 177 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835441:0:48445:0:(rw26.c:875:ll_write_end()) pos 724992, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835443:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000aeff356a[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 177] 00000080:00200000:2.0:1787827204.835445:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000030c81c47 17ffffc0001001 3:0 ffff8ab3c82a50e8 177 lru 00000080:00200000:2.0:1787827204.835448:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000ce74d804 177: 1< 0 - - > 2< 724992 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835450:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000aeff356a 00000080:00200000:2.0:1787827204.835450:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 178. 00000080:00200000:2.0:1787827204.835452:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 178 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835466:0:48445:0:(rw26.c:875:ll_write_end()) pos 729088, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835470:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000dd85f27a[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 178] 00000080:00200000:2.0:1787827204.835473:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000b4baffbb 17ffffc0001001 3:0 ffff8ab3c8c799f8 178 lru 00000080:00200000:2.0:1787827204.835479:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000e93dd7f2 178: 1< 0 - - > 2< 729088 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835482:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000dd85f27a 00000080:00200000:2.0:1787827204.835483:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 179. 00000080:00200000:2.0:1787827204.835485:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 179 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835501:0:48445:0:(rw26.c:875:ll_write_end()) pos 733184, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835505:0:48445:0:(rw26.c:933:ll_write_end()) page@000000003cee303e[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 179] 00000080:00200000:2.0:1787827204.835508:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000b423d727 17ffffc0001001 3:0 ffff8ab3c8c790e8 179 lru 00000080:00200000:2.0:1787827204.835515:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000007b9bdcf7 179: 1< 0 - - > 2< 733184 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835517:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000003cee303e 00000080:00200000:2.0:1787827204.835519:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 180. 00000080:00200000:2.0:1787827204.835521:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 180 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835539:0:48445:0:(rw26.c:875:ll_write_end()) pos 737280, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835544:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000450e4f40[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 180] 00000080:00200000:2.0:1787827204.835546:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000009b04c3e4 17ffffc0001001 3:0 ffff8ab3c8c79740 180 lru 00000080:00200000:2.0:1787827204.835552:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000e6a4bfb1 180: 1< 0 - - > 2< 737280 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835554:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000450e4f40 00000080:00200000:2.0:1787827204.835555:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 181. 00000080:00200000:2.0:1787827204.835557:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 181 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835572:0:48445:0:(rw26.c:875:ll_write_end()) pos 741376, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835577:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000cbeec30d[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 181] 00000080:00200000:2.0:1787827204.835580:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000aedc869f 17ffffc0001001 3:0 ffff8ab3c8c79570 181 lru 00000080:00200000:2.0:1787827204.835586:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000023025294 181: 1< 0 - - > 2< 741376 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835588:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000cbeec30d 00000080:00200000:2.0:1787827204.835589:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 182. 00000080:00200000:2.0:1787827204.835591:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 182 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835606:0:48445:0:(rw26.c:875:ll_write_end()) pos 745472, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835611:0:48445:0:(rw26.c:933:ll_write_end()) page@000000007907858a[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 182] 00000080:00200000:2.0:1787827204.835613:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000009fd5bb4e 17ffffc0001001 3:0 ffff8ab3c8c791d0 182 lru 00000080:00200000:2.0:1787827204.835619:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000b6548531 182: 1< 0 - - > 2< 745472 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835622:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000007907858a 00000080:00200000:2.0:1787827204.835623:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 183. 00000080:00200000:2.0:1787827204.835624:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 183 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835640:0:48445:0:(rw26.c:875:ll_write_end()) pos 749568, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835645:0:48445:0:(rw26.c:933:ll_write_end()) page@000000004344da83[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 183] 00000080:00200000:2.0:1787827204.835648:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000005d03fa4e 17ffffc0001001 3:0 ffff8ab3c8c79658 183 lru 00000080:00200000:2.0:1787827204.835655:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000058b3bbf2 183: 1< 0 - - > 2< 749568 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835657:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000004344da83 00000080:00200000:2.0:1787827204.835658:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 184. 00000080:00200000:2.0:1787827204.835660:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 184 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835674:0:48445:0:(rw26.c:875:ll_write_end()) pos 753664, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835678:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000ccb085f3[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 184] 00000080:00200000:2.0:1787827204.835681:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000befdba95 17ffffc0001001 3:0 ffff8ab3c8c79000 184 lru 00000080:00200000:2.0:1787827204.835687:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000007b0bf5e6 184: 1< 0 - - > 2< 753664 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835690:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000ccb085f3 00000080:00200000:2.0:1787827204.835691:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 185. 00000080:00200000:2.0:1787827204.835693:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 185 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835705:0:48445:0:(rw26.c:875:ll_write_end()) pos 757760, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835708:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000017a6908f[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 185] 00000080:00200000:2.0:1787827204.835710:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000026853ec0 17ffffc0001001 3:0 ffff8ab3c8c79e80 185 lru 00000080:00200000:2.0:1787827204.835714:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000bbbfd74d 185: 1< 0 - - > 2< 757760 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835715:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000017a6908f 00000080:00200000:2.0:1787827204.835716:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 186. 00000080:00200000:2.0:1787827204.835718:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 186 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835726:0:48445:0:(rw26.c:875:ll_write_end()) pos 761856, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835729:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000015b88e7[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 186] 00000080:00200000:2.0:1787827204.835731:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000549d8172 17ffffc0001001 3:0 ffff8ab3c8c79ae0 186 lru 00000080:00200000:2.0:1787827204.835735:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000018d746d8 186: 1< 0 - - > 2< 761856 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835738:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000015b88e7 00000080:00200000:2.0:1787827204.835739:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 187. 00000080:00200000:2.0:1787827204.835741:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 187 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835755:0:48445:0:(rw26.c:875:ll_write_end()) pos 765952, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835760:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000073189560[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 187] 00000080:00200000:2.0:1787827204.835763:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000060b54354 17ffffc0001001 3:0 ffff8ab3c8c79bc8 187 lru 00000080:00200000:2.0:1787827204.835768:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000003f87a7a0 187: 1< 0 - - > 2< 765952 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835771:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000073189560 00000080:00200000:2.0:1787827204.835772:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 188. 00000080:00200000:2.0:1787827204.835774:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 188 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835789:0:48445:0:(rw26.c:875:ll_write_end()) pos 770048, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835793:0:48445:0:(rw26.c:933:ll_write_end()) page@000000005f6184eb[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 188] 00000080:00200000:2.0:1787827204.835796:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000dd6bf7e2 17ffffc0001001 3:0 ffff8ab3c8c79910 188 lru 00000080:00200000:2.0:1787827204.835802:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000080cf9541 188: 1< 0 - - > 2< 770048 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835804:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000005f6184eb 00000080:00200000:2.0:1787827204.835806:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 189. 00000080:00200000:2.0:1787827204.835808:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 189 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835829:0:48445:0:(rw26.c:875:ll_write_end()) pos 774144, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835834:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000f8c14848[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 189] 00000080:00200000:2.0:1787827204.835837:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000007b7c9e33 17ffffc0001021 2:0 ffff8ab3c8c79828 189 lru 00000080:00200000:2.0:1787827204.835843:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000a47ea235 189: 1< 0 - - > 2< 774144 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835846:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000f8c14848 00000080:00200000:2.0:1787827204.835847:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 190. 00000080:00200000:2.0:1787827204.835849:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 190 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835863:0:48445:0:(rw26.c:875:ll_write_end()) pos 778240, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835868:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000018505cc0[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 190] 00000080:00200000:2.0:1787827204.835870:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000491d315f 17ffffc0001001 3:0 ffff8ab3c8c79cb0 190 lru 00000080:00200000:2.0:1787827204.835876:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000761110d0 190: 1< 0 - - > 2< 778240 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835879:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000018505cc0 00000080:00200000:2.0:1787827204.835880:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 191. 00000080:00200000:2.0:1787827204.835882:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 191 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835918:0:48445:0:(rw26.c:875:ll_write_end()) pos 782336, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835923:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000080359dc[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 191] 00000080:00200000:2.0:1787827204.835927:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000ec563386 17ffffc0001001 3:0 ffff8ab3c8c79d98 191 lru 00000080:00200000:2.0:1787827204.835932:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000002fa0d801 191: 1< 0 - - > 2< 782336 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835934:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000080359dc 00000080:00200000:2.0:1787827204.835935:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 192. 00000080:00200000:2.0:1787827204.835938:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 192 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.835956:0:48445:0:(rw26.c:875:ll_write_end()) pos 786432, len 4096, copied 4096 00000080:00200000:2.0:1787827204.835961:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000e42c4505[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 192] 00000080:00200000:2.0:1787827204.835974:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000081e28819 17ffffc0001001 3:0 ffff8ab3c8c792b8 192 lru 00000080:00200000:2.0:1787827204.835980:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000bd6826b7 192: 1< 0 - - > 2< 786432 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.835982:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000e42c4505 00000080:00200000:2.0:1787827204.835983:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 193. 00000080:00200000:2.0:1787827204.835985:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 193 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836000:0:48445:0:(rw26.c:875:ll_write_end()) pos 790528, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836005:0:48445:0:(rw26.c:933:ll_write_end()) page@000000001545d496[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 193] 00000080:00200000:2.0:1787827204.836008:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000001db39129 17ffffc0001001 3:0 ffff8ab3c8c79488 193 lru 00000080:00200000:2.0:1787827204.836014:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000029ab3f1 193: 1< 0 - - > 2< 790528 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836017:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000001545d496 00000080:00200000:2.0:1787827204.836019:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 194. 00000080:00200000:2.0:1787827204.836021:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 194 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836050:0:48445:0:(rw26.c:875:ll_write_end()) pos 794624, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836054:0:48445:0:(rw26.c:933:ll_write_end()) page@000000003deba6f0[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 194] 00000080:00200000:2.0:1787827204.836057:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000c3ec1673 17ffffc0001001 3:0 ffff8ab3c8c793a0 194 lru 00000080:00200000:2.0:1787827204.836063:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000084fbf8b6 194: 1< 0 - - > 2< 794624 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836065:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000003deba6f0 00000080:00200000:2.0:1787827204.836067:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 195. 00000080:00200000:2.0:1787827204.836069:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 195 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836091:0:48445:0:(rw26.c:875:ll_write_end()) pos 798720, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836110:0:48445:0:(rw26.c:933:ll_write_end()) page@000000004ff9db52[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 195] 00000080:00200000:2.0:1787827204.836113:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000008095e7f5 17ffffc0001001 3:0 ffff8ab3f062b0e8 195 lru 00000080:00200000:2.0:1787827204.836119:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000fcd061a0 195: 1< 0 - - > 2< 798720 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836122:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000004ff9db52 00000080:00200000:2.0:1787827204.836124:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 196. 00000080:00200000:2.0:1787827204.836126:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 196 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836144:0:48445:0:(rw26.c:875:ll_write_end()) pos 802816, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836148:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000122e3506[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 196] 00000080:00200000:2.0:1787827204.836151:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000c61efafe 17ffffc0001001 3:0 ffff8ab3f062b740 196 lru 00000080:00200000:2.0:1787827204.836157:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000000d99f4f0 196: 1< 0 - - > 2< 802816 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836160:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000122e3506 00000080:00200000:2.0:1787827204.836161:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 197. 00000080:00200000:2.0:1787827204.836163:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 197 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836179:0:48445:0:(rw26.c:875:ll_write_end()) pos 806912, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836184:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000af47e727[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 197] 00000080:00200000:2.0:1787827204.836187:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000c3060d5b 17ffffc0001001 3:0 ffff8ab3f062b570 197 lru 00000080:00200000:2.0:1787827204.836193:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000007012ec8d 197: 1< 0 - - > 2< 806912 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836195:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000af47e727 00000080:00200000:2.0:1787827204.836196:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 198. 00000080:00200000:2.0:1787827204.836199:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 198 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836215:0:48445:0:(rw26.c:875:ll_write_end()) pos 811008, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836220:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000b181b891[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 198] 00000080:00200000:2.0:1787827204.836223:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000a79fd14e 17ffffc0001001 3:0 ffff8ab3f062b1d0 198 lru 00000080:00200000:2.0:1787827204.836229:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000a25a1759 198: 1< 0 - - > 2< 811008 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836232:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000b181b891 00000080:00200000:2.0:1787827204.836233:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 199. 00000080:00200000:2.0:1787827204.836235:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 199 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836253:0:48445:0:(rw26.c:875:ll_write_end()) pos 815104, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836258:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000092d525cf[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 199] 00000080:00200000:2.0:1787827204.836260:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000728bd4d8 17ffffc0001001 3:0 ffff8ab3f062b658 199 lru 00000080:00200000:2.0:1787827204.836266:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000002662f553 199: 1< 0 - - > 2< 815104 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836268:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000092d525cf 00000080:00200000:2.0:1787827204.836269:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 200. 00000080:00200000:2.0:1787827204.836271:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 200 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836286:0:48445:0:(rw26.c:875:ll_write_end()) pos 819200, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836291:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000f5b4ed16[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 200] 00000080:00200000:2.0:1787827204.836294:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000017d7fb48 17ffffc0001001 3:0 ffff8ab3f062b000 200 lru 00000080:00200000:2.0:1787827204.836300:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000060ce8688 200: 1< 0 - - > 2< 819200 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836302:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000f5b4ed16 00000080:00200000:2.0:1787827204.836303:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 201. 00000080:00200000:2.0:1787827204.836305:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 201 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836321:0:48445:0:(rw26.c:875:ll_write_end()) pos 823296, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836326:0:48445:0:(rw26.c:933:ll_write_end()) page@000000004560c33b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 201] 00000080:00200000:2.0:1787827204.836329:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000006227abf1 17ffffc0001001 3:0 ffff8ab3f062be80 201 lru 00000080:00200000:2.0:1787827204.836335:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000009fb676e0 201: 1< 0 - - > 2< 823296 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836338:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000004560c33b 00000080:00200000:2.0:1787827204.836339:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 202. 00000080:00200000:2.0:1787827204.836341:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 202 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836355:0:48445:0:(rw26.c:875:ll_write_end()) pos 827392, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836360:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000ef443b08[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 202] 00000080:00200000:2.0:1787827204.836364:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000002741c719 17ffffc0001001 3:0 ffff8ab3f062bae0 202 lru 00000080:00200000:2.0:1787827204.836370:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000f3a1cead 202: 1< 0 - - > 2< 827392 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836373:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000ef443b08 00000080:00200000:2.0:1787827204.836375:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 203. 00000080:00200000:2.0:1787827204.836377:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 203 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836392:0:48445:0:(rw26.c:875:ll_write_end()) pos 831488, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836397:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000a83ec40b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 203] 00000080:00200000:2.0:1787827204.836400:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000e1486786 17ffffc0001001 3:0 ffff8ab3f062bbc8 203 lru 00000080:00200000:2.0:1787827204.836406:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000002b3c88eb 203: 1< 0 - - > 2< 831488 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836408:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000a83ec40b 00000080:00200000:2.0:1787827204.836409:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 204. 00000080:00200000:2.0:1787827204.836411:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 204 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836431:0:48445:0:(rw26.c:875:ll_write_end()) pos 835584, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836435:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000b831c2b5[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 204] 00000080:00200000:2.0:1787827204.836438:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000585daca0 17ffffc0001021 2:0 ffff8ab3f062b910 204 lru 00000080:00200000:2.0:1787827204.836444:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000e775e748 204: 1< 0 - - > 2< 835584 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836446:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000b831c2b5 00000080:00200000:2.0:1787827204.836447:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 205. 00000080:00200000:2.0:1787827204.836450:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 205 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836466:0:48445:0:(rw26.c:875:ll_write_end()) pos 839680, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836470:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000ec548155[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 205] 00000080:00200000:2.0:1787827204.836473:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000b5c12b5d 17ffffc0001001 3:0 ffff8ab3f062b828 205 lru 00000080:00200000:2.0:1787827204.836479:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000009d210022 205: 1< 0 - - > 2< 839680 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836482:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000ec548155 00000080:00200000:2.0:1787827204.836483:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 206. 00000080:00200000:2.0:1787827204.836486:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 206 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836501:0:48445:0:(rw26.c:875:ll_write_end()) pos 843776, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836506:0:48445:0:(rw26.c:933:ll_write_end()) page@000000005ed60613[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 206] 00000080:00200000:2.0:1787827204.836509:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000001b47914f 17ffffc0001001 3:0 ffff8ab3f062bcb0 206 lru 00000080:00200000:2.0:1787827204.836515:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000a9348b72 206: 1< 0 - - > 2< 843776 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836517:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000005ed60613 00000080:00200000:2.0:1787827204.836519:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 207. 00000080:00200000:2.0:1787827204.836521:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 207 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836537:0:48445:0:(rw26.c:875:ll_write_end()) pos 847872, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836542:0:48445:0:(rw26.c:933:ll_write_end()) page@000000003283508c[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 207] 00000080:00200000:2.0:1787827204.836545:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000000bb15876 17ffffc0001001 3:0 ffff8ab3f062bd98 207 lru 00000080:00200000:2.0:1787827204.836552:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000a695648d 207: 1< 0 - - > 2< 847872 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836554:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000003283508c 00000080:00200000:2.0:1787827204.836556:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 208. 00000080:00200000:2.0:1787827204.836558:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 208 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836576:0:48445:0:(rw26.c:875:ll_write_end()) pos 851968, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836582:0:48445:0:(rw26.c:933:ll_write_end()) page@000000002242c3e5[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 208] 00000080:00200000:2.0:1787827204.836585:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000167a63d2 17ffffc0001001 3:0 ffff8ab3f062b2b8 208 lru 00000080:00200000:2.0:1787827204.836592:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000089e189f 208: 1< 0 - - > 2< 851968 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836594:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000002242c3e5 00000080:00200000:2.0:1787827204.836595:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 209. 00000080:00200000:2.0:1787827204.836598:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 209 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836613:0:48445:0:(rw26.c:875:ll_write_end()) pos 856064, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836618:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000e6a5398e[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 209] 00000080:00200000:2.0:1787827204.836621:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000fb19d87f 17ffffc0001001 3:0 ffff8ab3f062b488 209 lru 00000080:00200000:2.0:1787827204.836628:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000009b2e0c71 209: 1< 0 - - > 2< 856064 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836630:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000e6a5398e 00000080:00200000:2.0:1787827204.836631:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 210. 00000080:00200000:2.0:1787827204.836633:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 210 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836650:0:48445:0:(rw26.c:875:ll_write_end()) pos 860160, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836655:0:48445:0:(rw26.c:933:ll_write_end()) page@000000007c0c68b9[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 210] 00000080:00200000:2.0:1787827204.836658:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000c00eedec 17ffffc0001001 3:0 ffff8ab3f062b3a0 210 lru 00000080:00200000:2.0:1787827204.836664:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000567d7508 210: 1< 0 - - > 2< 860160 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836667:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000007c0c68b9 00000080:00200000:2.0:1787827204.836668:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 211. 00000080:00200000:2.0:1787827204.836671:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 211 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836686:0:48445:0:(rw26.c:875:ll_write_end()) pos 864256, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836691:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000b013437e[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 211] 00000080:00200000:2.0:1787827204.836694:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000bbba6792 17ffffc0001001 3:0 ffff8ab3f062b9f8 211 lru 00000080:00200000:2.0:1787827204.836700:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000036d86204 211: 1< 0 - - > 2< 864256 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836703:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000b013437e 00000080:00200000:2.0:1787827204.836704:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 212. 00000080:00200000:2.0:1787827204.836706:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 212 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836730:0:48445:0:(rw26.c:875:ll_write_end()) pos 868352, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836735:0:48445:0:(rw26.c:933:ll_write_end()) page@000000009ba8b650[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 212] 00000080:00200000:2.0:1787827204.836739:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000a50403bc 17ffffc0001001 3:0 ffff8ab3c4969488 212 lru 00000080:00200000:2.0:1787827204.836745:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000ff68e876 212: 1< 0 - - > 2< 868352 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836748:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000009ba8b650 00000080:00200000:2.0:1787827204.836749:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 213. 00000080:00200000:2.0:1787827204.836751:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 213 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836768:0:48445:0:(rw26.c:875:ll_write_end()) pos 872448, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836773:0:48445:0:(rw26.c:933:ll_write_end()) page@000000008b81369d[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 213] 00000080:00200000:2.0:1787827204.836776:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000066c854ed 17ffffc0001001 3:0 ffff8ab3c49693a0 213 lru 00000080:00200000:2.0:1787827204.836783:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000000cf922fb 213: 1< 0 - - > 2< 872448 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836785:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000008b81369d 00000080:00200000:2.0:1787827204.836787:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 214. 00000080:00200000:2.0:1787827204.836789:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 214 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836806:0:48445:0:(rw26.c:875:ll_write_end()) pos 876544, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836810:0:48445:0:(rw26.c:933:ll_write_end()) page@000000000124f581[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 214] 00000080:00200000:2.0:1787827204.836814:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000f5608509 17ffffc0001001 3:0 ffff8ab3c49699f8 214 lru 00000080:00200000:2.0:1787827204.836820:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000d10e0e7d 214: 1< 0 - - > 2< 876544 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836823:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000000124f581 00000080:00200000:2.0:1787827204.836824:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 215. 00000080:00200000:2.0:1787827204.836826:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 215 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836845:0:48445:0:(rw26.c:875:ll_write_end()) pos 880640, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836850:0:48445:0:(rw26.c:933:ll_write_end()) page@000000008764c1f4[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 215] 00000080:00200000:2.0:1787827204.836853:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000e95c9a41 17ffffc0001001 3:0 ffff8ab3c49690e8 215 lru 00000080:00200000:2.0:1787827204.836859:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000861cbb42 215: 1< 0 - - > 2< 880640 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836861:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000008764c1f4 00000080:00200000:2.0:1787827204.836863:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 216. 00000080:00200000:2.0:1787827204.836865:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 216 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836881:0:48445:0:(rw26.c:875:ll_write_end()) pos 884736, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836906:0:48445:0:(rw26.c:933:ll_write_end()) page@000000000fd1e07a[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 216] 00000080:00200000:2.0:1787827204.836910:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000003f25951d 17ffffc0001001 3:0 ffff8ab3c4969740 216 lru 00000080:00200000:2.0:1787827204.836916:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000adf2212a 216: 1< 0 - - > 2< 884736 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836919:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000000fd1e07a 00000080:00200000:2.0:1787827204.836920:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 217. 00000080:00200000:2.0:1787827204.836923:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 217 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836953:0:48445:0:(rw26.c:875:ll_write_end()) pos 888832, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836958:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000bba53aa0[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 217] 00000080:00200000:2.0:1787827204.836962:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000e69d683c 17ffffc0001001 3:0 ffff8ab3c4969570 217 lru 00000080:00200000:2.0:1787827204.836968:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000cf1278fa 217: 1< 0 - - > 2< 888832 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.836970:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000bba53aa0 00000080:00200000:2.0:1787827204.836971:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 218. 00000080:00200000:2.0:1787827204.836974:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 218 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.836989:0:48445:0:(rw26.c:875:ll_write_end()) pos 892928, len 4096, copied 4096 00000080:00200000:2.0:1787827204.836994:0:48445:0:(rw26.c:933:ll_write_end()) page@000000002c15637b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 218] 00000080:00200000:2.0:1787827204.836998:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000d71002f1 17ffffc0001001 3:0 ffff8ab3c49691d0 218 lru 00000080:00200000:2.0:1787827204.837004:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000e188c0a6 218: 1< 0 - - > 2< 892928 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837006:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000002c15637b 00000080:00200000:2.0:1787827204.837007:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 219. 00000080:00200000:2.0:1787827204.837010:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 219 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837029:0:48445:0:(rw26.c:875:ll_write_end()) pos 897024, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837035:0:48445:0:(rw26.c:933:ll_write_end()) page@000000000d838a18[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 219] 00000080:00200000:2.0:1787827204.837038:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000f499c140 17ffffc0001021 2:0 ffff8ab3c4969658 219 lru 00000080:00200000:2.0:1787827204.837054:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000dc83271c 219: 1< 0 - - > 2< 897024 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837057:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000000d838a18 00000080:00200000:2.0:1787827204.837058:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 220. 00000080:00200000:2.0:1787827204.837061:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 220 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837077:0:48445:0:(rw26.c:875:ll_write_end()) pos 901120, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837083:0:48445:0:(rw26.c:933:ll_write_end()) page@000000002f383850[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 220] 00000080:00200000:2.0:1787827204.837086:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000530fec5e 17ffffc0001001 3:0 ffff8ab3c4969000 220 lru 00000080:00200000:2.0:1787827204.837107:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000004431c547 220: 1< 0 - - > 2< 901120 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837111:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000002f383850 00000080:00200000:2.0:1787827204.837113:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 221. 00000080:00200000:2.0:1787827204.837115:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 221 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837132:0:48445:0:(rw26.c:875:ll_write_end()) pos 905216, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837137:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000054d44385[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 221] 00000080:00200000:2.0:1787827204.837140:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000415a92c5 17ffffc0001001 3:0 ffff8ab3c4969e80 221 lru 00000080:00200000:2.0:1787827204.837146:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000004f17d338 221: 1< 0 - - > 2< 905216 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837149:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000054d44385 00000080:00200000:2.0:1787827204.837150:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 222. 00000080:00200000:2.0:1787827204.837152:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 222 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837170:0:48445:0:(rw26.c:875:ll_write_end()) pos 909312, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837175:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000bff45ed0[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 222] 00000080:00200000:2.0:1787827204.837178:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000004f84e7d3 17ffffc0001001 3:0 ffff8ab3c4969ae0 222 lru 00000080:00200000:2.0:1787827204.837184:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000fd3c9329 222: 1< 0 - - > 2< 909312 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837187:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000bff45ed0 00000080:00200000:2.0:1787827204.837188:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 223. 00000080:00200000:2.0:1787827204.837190:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 223 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837207:0:48445:0:(rw26.c:875:ll_write_end()) pos 913408, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837212:0:48445:0:(rw26.c:933:ll_write_end()) page@000000005bcb092f[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 223] 00000080:00200000:2.0:1787827204.837215:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000465ce536 17ffffc0001001 3:0 ffff8ab3c4969bc8 223 lru 00000080:00200000:2.0:1787827204.837222:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000008eea9b25 223: 1< 0 - - > 2< 913408 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837224:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000005bcb092f 00000080:00200000:2.0:1787827204.837226:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 224. 00000080:00200000:2.0:1787827204.837228:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 224 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837245:0:48445:0:(rw26.c:875:ll_write_end()) pos 917504, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837251:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000752aa4ca[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 224] 00000080:00200000:2.0:1787827204.837254:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000005edf87a0 17ffffc0001001 3:0 ffff8ab3c4969910 224 lru 00000080:00200000:2.0:1787827204.837260:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000004b11ed5f 224: 1< 0 - - > 2< 917504 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837263:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000752aa4ca 00000080:00200000:2.0:1787827204.837265:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 225. 00000080:00200000:2.0:1787827204.837266:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 225 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837282:0:48445:0:(rw26.c:875:ll_write_end()) pos 921600, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837287:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000bd57fcf8[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 225] 00000080:00200000:2.0:1787827204.837290:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000bf64943c 17ffffc0001001 3:0 ffff8ab3c4969828 225 lru 00000080:00200000:2.0:1787827204.837297:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000063a9cec6 225: 1< 0 - - > 2< 921600 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837299:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000bd57fcf8 00000080:00200000:2.0:1787827204.837300:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 226. 00000080:00200000:2.0:1787827204.837303:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 226 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837320:0:48445:0:(rw26.c:875:ll_write_end()) pos 925696, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837325:0:48445:0:(rw26.c:933:ll_write_end()) page@000000005bf48f7e[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 226] 00000080:00200000:2.0:1787827204.837329:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000116eb6bc 17ffffc0001001 3:0 ffff8ab3c4969cb0 226 lru 00000080:00200000:2.0:1787827204.837334:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000017256c95 226: 1< 0 - - > 2< 925696 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837337:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000005bf48f7e 00000080:00200000:2.0:1787827204.837338:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 227. 00000080:00200000:2.0:1787827204.837340:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 227 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837357:0:48445:0:(rw26.c:875:ll_write_end()) pos 929792, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837362:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000ba21037d[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 227] 00000080:00200000:2.0:1787827204.837365:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000002cb11351 17ffffc0001001 3:0 ffff8ab3c4969d98 227 lru 00000080:00200000:2.0:1787827204.837371:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000eadaa325 227: 1< 0 - - > 2< 929792 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837374:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000ba21037d 00000080:00200000:2.0:1787827204.837375:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 228. 00000080:00200000:2.0:1787827204.837377:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 228 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837392:0:48445:0:(rw26.c:875:ll_write_end()) pos 933888, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837397:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000fabad034[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 228] 00000080:00200000:2.0:1787827204.837401:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000019d9e6d3 17ffffc0001001 3:0 ffff8ab3c49692b8 228 lru 00000080:00200000:2.0:1787827204.837407:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000009999ff4 228: 1< 0 - - > 2< 933888 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837409:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000fabad034 00000080:00200000:2.0:1787827204.837411:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 229. 00000080:00200000:2.0:1787827204.837413:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 229 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837440:0:48445:0:(rw26.c:875:ll_write_end()) pos 937984, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837446:0:48445:0:(rw26.c:933:ll_write_end()) page@000000005a143105[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 229] 00000080:00200000:2.0:1787827204.837449:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000781a8084 17ffffc0001001 3:0 ffff8ab3d063eae0 229 lru 00000080:00200000:2.0:1787827204.837455:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000cea61e21 229: 1< 0 - - > 2< 937984 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837458:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000005a143105 00000080:00200000:2.0:1787827204.837459:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 230. 00000080:00200000:2.0:1787827204.837462:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 230 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837479:0:48445:0:(rw26.c:875:ll_write_end()) pos 942080, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837483:0:48445:0:(rw26.c:933:ll_write_end()) page@000000005da31cfe[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 230] 00000080:00200000:2.0:1787827204.837486:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000e5db171c 17ffffc0001001 3:0 ffff8ab3d063ebc8 230 lru 00000080:00200000:2.0:1787827204.837493:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000005fda2f5b 230: 1< 0 - - > 2< 942080 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837495:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000005da31cfe 00000080:00200000:2.0:1787827204.837497:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 231. 00000080:00200000:2.0:1787827204.837499:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 231 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837515:0:48445:0:(rw26.c:875:ll_write_end()) pos 946176, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837519:0:48445:0:(rw26.c:933:ll_write_end()) page@000000000b679f43[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 231] 00000080:00200000:2.0:1787827204.837522:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000d3efc689 17ffffc0001001 3:0 ffff8ab3d063e910 231 lru 00000080:00200000:2.0:1787827204.837528:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000a36eba3b 231: 1< 0 - - > 2< 946176 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837531:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000000b679f43 00000080:00200000:2.0:1787827204.837532:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 232. 00000080:00200000:2.0:1787827204.837534:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 232 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837547:0:48445:0:(rw26.c:875:ll_write_end()) pos 950272, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837551:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000f4f0ce0a[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 232] 00000080:00200000:2.0:1787827204.837553:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000009c6d17d5 17ffffc0001001 3:0 ffff8ab3d063e828 232 lru 00000080:00200000:2.0:1787827204.837559:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000099c593d9 232: 1< 0 - - > 2< 950272 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837561:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000f4f0ce0a 00000080:00200000:2.0:1787827204.837563:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 233. 00000080:00200000:2.0:1787827204.837565:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 233 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837576:0:48445:0:(rw26.c:875:ll_write_end()) pos 954368, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837581:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000166bbf70[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 233] 00000080:00200000:2.0:1787827204.837583:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000006a1993b1 17ffffc0001001 3:0 ffff8ab3d063ecb0 233 lru 00000080:00200000:2.0:1787827204.837589:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000b902982d 233: 1< 0 - - > 2< 954368 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837591:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000166bbf70 00000080:00200000:2.0:1787827204.837592:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 234. 00000080:00200000:2.0:1787827204.837594:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 234 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837610:0:48445:0:(rw26.c:875:ll_write_end()) pos 958464, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837614:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000d69978b0[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 234] 00000080:00200000:2.0:1787827204.837616:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000003d21e323 17ffffc0001021 2:0 ffff8ab3d063ed98 234 lru 00000080:00200000:2.0:1787827204.837622:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000058afadca 234: 1< 0 - - > 2< 958464 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837624:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000d69978b0 00000080:00200000:2.0:1787827204.837626:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 235. 00000080:00200000:2.0:1787827204.837627:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 235 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837640:0:48445:0:(rw26.c:875:ll_write_end()) pos 962560, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837644:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000e0a5f14e[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 235] 00000080:00200000:2.0:1787827204.837646:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000d8f9b9e6 17ffffc0001001 3:0 ffff8ab3d063e2b8 235 lru 00000080:00200000:2.0:1787827204.837651:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000000e1e54cf 235: 1< 0 - - > 2< 962560 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837654:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000e0a5f14e 00000080:00200000:2.0:1787827204.837655:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 236. 00000080:00200000:2.0:1787827204.837657:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 236 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837671:0:48445:0:(rw26.c:875:ll_write_end()) pos 966656, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837675:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000a8543f2c[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 236] 00000080:00200000:2.0:1787827204.837679:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000bffd6567 17ffffc0001001 3:0 ffff8ab3d063e488 236 lru 00000080:00200000:2.0:1787827204.837684:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000052bc7b5b 236: 1< 0 - - > 2< 966656 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837686:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000a8543f2c 00000080:00200000:2.0:1787827204.837687:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 237. 00000080:00200000:2.0:1787827204.837689:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 237 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837702:0:48445:0:(rw26.c:875:ll_write_end()) pos 970752, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837706:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000af97807f[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 237] 00000080:00200000:2.0:1787827204.837709:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000094972bba 17ffffc0001001 3:0 ffff8ab3d063e3a0 237 lru 00000080:00200000:2.0:1787827204.837714:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000e811ad03 237: 1< 0 - - > 2< 970752 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837716:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000af97807f 00000080:00200000:2.0:1787827204.837718:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 238. 00000080:00200000:2.0:1787827204.837719:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 238 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837732:0:48445:0:(rw26.c:875:ll_write_end()) pos 974848, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837736:0:48445:0:(rw26.c:933:ll_write_end()) page@000000005a302e27[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 238] 00000080:00200000:2.0:1787827204.837739:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000e3c79037 17ffffc0001001 3:0 ffff8ab3d063e9f8 238 lru 00000080:00200000:2.0:1787827204.837744:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000001c336de3 238: 1< 0 - - > 2< 974848 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837747:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000005a302e27 00000080:00200000:2.0:1787827204.837748:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 239. 00000080:00200000:2.0:1787827204.837750:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 239 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837762:0:48445:0:(rw26.c:875:ll_write_end()) pos 978944, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837766:0:48445:0:(rw26.c:933:ll_write_end()) page@000000003d7a42bc[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 239] 00000080:00200000:2.0:1787827204.837769:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000003355f89b 17ffffc0001001 3:0 ffff8ab3d063e0e8 239 lru 00000080:00200000:2.0:1787827204.837774:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000ec3ab337 239: 1< 0 - - > 2< 978944 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837777:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000003d7a42bc 00000080:00200000:2.0:1787827204.837778:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 240. 00000080:00200000:2.0:1787827204.837780:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 240 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837794:0:48445:0:(rw26.c:875:ll_write_end()) pos 983040, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837798:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000a88932c4[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 240] 00000080:00200000:2.0:1787827204.837801:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000002c36235e 17ffffc0001001 3:0 ffff8ab3d063e740 240 lru 00000080:00200000:2.0:1787827204.837806:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000034d12338 240: 1< 0 - - > 2< 983040 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837808:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000a88932c4 00000080:00200000:2.0:1787827204.837810:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 241. 00000080:00200000:2.0:1787827204.837812:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 241 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837825:0:48445:0:(rw26.c:875:ll_write_end()) pos 987136, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837830:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000e0cc7f04[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 241] 00000080:00200000:2.0:1787827204.837833:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000edce15ca 17ffffc0001001 3:0 ffff8ab3d063e570 241 lru 00000080:00200000:2.0:1787827204.837838:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000013ac3974 241: 1< 0 - - > 2< 987136 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837840:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000e0cc7f04 00000080:00200000:2.0:1787827204.837842:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 242. 00000080:00200000:2.0:1787827204.837844:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 242 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837858:0:48445:0:(rw26.c:875:ll_write_end()) pos 991232, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837862:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000f0b53c9e[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 242] 00000080:00200000:2.0:1787827204.837865:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000618ef2d6 17ffffc0001001 3:0 ffff8ab3d063e1d0 242 lru 00000080:00200000:2.0:1787827204.837871:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000ec7fe505 242: 1< 0 - - > 2< 991232 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837874:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000f0b53c9e 00000080:00200000:2.0:1787827204.837875:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 243. 00000080:00200000:2.0:1787827204.837877:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 243 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837914:0:48445:0:(rw26.c:875:ll_write_end()) pos 995328, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837921:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000023674918[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 243] 00000080:00200000:2.0:1787827204.837925:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000914388c6 17ffffc0001001 3:0 ffff8ab3d063e658 243 lru 00000080:00200000:2.0:1787827204.837931:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000ad4e9475 243: 1< 0 - - > 2< 995328 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837934:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000023674918 00000080:00200000:2.0:1787827204.837936:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 244. 00000080:00200000:2.0:1787827204.837939:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 244 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837956:0:48445:0:(rw26.c:875:ll_write_end()) pos 999424, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837961:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000ab87bc69[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 244] 00000080:00200000:2.0:1787827204.837964:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000f18507ae 17ffffc0001001 3:0 ffff8ab3d063e000 244 lru 00000080:00200000:2.0:1787827204.837970:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000b8a5dd40 244: 1< 0 - - > 2< 999424 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.837973:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000ab87bc69 00000080:00200000:2.0:1787827204.837974:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 245. 00000080:00200000:2.0:1787827204.837976:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 245 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.837991:0:48445:0:(rw26.c:875:ll_write_end()) pos 1003520, len 4096, copied 4096 00000080:00200000:2.0:1787827204.837996:0:48445:0:(rw26.c:933:ll_write_end()) page@000000005a3f2f9b[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 245] 00000080:00200000:2.0:1787827204.837999:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000009fbf3cc 17ffffc0001001 3:0 ffff8ab3d063ee80 245 lru 00000080:00200000:2.0:1787827204.838005:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000003d3f2179 245: 1< 0 - - > 2< 1003520 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.838008:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000005a3f2f9b 00000080:00200000:2.0:1787827204.838009:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 246. 00000080:00200000:2.0:1787827204.838011:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 246 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.838036:0:48445:0:(rw26.c:875:ll_write_end()) pos 1007616, len 4096, copied 4096 00000080:00200000:2.0:1787827204.838041:0:48445:0:(rw26.c:933:ll_write_end()) page@000000009b7e7df2[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 246] 00000080:00200000:2.0:1787827204.838045:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000007095c9c7 17ffffc0001001 3:0 ffff8ab3e0961570 246 lru 00000080:00200000:2.0:1787827204.838050:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000112d39ae 246: 1< 0 - - > 2< 1007616 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.838053:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000009b7e7df2 00000080:00200000:2.0:1787827204.838054:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 247. 00000080:00200000:2.0:1787827204.838056:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 247 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.838072:0:48445:0:(rw26.c:875:ll_write_end()) pos 1011712, len 4096, copied 4096 00000080:00200000:2.0:1787827204.838077:0:48445:0:(rw26.c:933:ll_write_end()) page@000000006dd1d440[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 247] 00000080:00200000:2.0:1787827204.838080:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000821bef78 17ffffc0001001 3:0 ffff8ab3e09611d0 247 lru 00000080:00200000:2.0:1787827204.838085:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000005d5c277e 247: 1< 0 - - > 2< 1011712 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.838088:0:48445:0:(rw26.c:933:ll_write_end()) end page@000000006dd1d440 00000080:00200000:2.0:1787827204.838090:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 248. 00000080:00200000:2.0:1787827204.838092:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 248 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.838125:0:48445:0:(rw26.c:875:ll_write_end()) pos 1015808, len 4096, copied 4096 00000080:00200000:2.0:1787827204.838130:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000a4575413[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 248] 00000080:00200000:2.0:1787827204.838134:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000018081820 17ffffc0001001 3:0 ffff8ab3e0961658 248 lru 00000080:00200000:2.0:1787827204.838139:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000034558b33 248: 1< 0 - - > 2< 1015808 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.838142:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000a4575413 00000080:00200000:2.0:1787827204.838143:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 249. 00000080:00200000:2.0:1787827204.838145:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 249 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.838164:0:48445:0:(rw26.c:875:ll_write_end()) pos 1019904, len 4096, copied 4096 00000080:00200000:2.0:1787827204.838168:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000383f3d22[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 249] 00000080:00200000:2.0:1787827204.838171:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000004f108a49 17ffffc0001021 2:0 ffff8ab3e0961000 249 lru 00000080:00200000:2.0:1787827204.838178:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000008984db07 249: 1< 0 - - > 2< 1019904 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.838180:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000383f3d22 00000080:00200000:2.0:1787827204.838182:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 250. 00000080:00200000:2.0:1787827204.838184:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 250 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.838198:0:48445:0:(rw26.c:875:ll_write_end()) pos 1024000, len 4096, copied 4096 00000080:00200000:2.0:1787827204.838202:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000d4f7dcc8[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 250] 00000080:00200000:2.0:1787827204.838206:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000f4067484 17ffffc0001001 3:0 ffff8ab3e0961e80 250 lru 00000080:00200000:2.0:1787827204.838211:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@0000000036e263cc 250: 1< 0 - - > 2< 1024000 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.838213:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000d4f7dcc8 00000080:00200000:2.0:1787827204.838215:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 251. 00000080:00200000:2.0:1787827204.838217:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 251 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.838231:0:48445:0:(rw26.c:875:ll_write_end()) pos 1028096, len 4096, copied 4096 00000080:00200000:2.0:1787827204.838236:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000032c097fe[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 251] 00000080:00200000:2.0:1787827204.838240:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @0000000017f691f4 17ffffc0001001 3:0 ffff8ab3e0961ae0 251 lru 00000080:00200000:2.0:1787827204.838246:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000009c9d30a8 251: 1< 0 - - > 2< 1028096 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.838248:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000032c097fe 00000080:00200000:2.0:1787827204.838250:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 252. 00000080:00200000:2.0:1787827204.838252:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 252 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.838267:0:48445:0:(rw26.c:875:ll_write_end()) pos 1032192, len 4096, copied 4096 00000080:00200000:2.0:1787827204.838272:0:48445:0:(rw26.c:933:ll_write_end()) page@0000000026c57f9f[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 252] 00000080:00200000:2.0:1787827204.838275:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000b90322ba 17ffffc0001001 3:0 ffff8ab3e0961bc8 252 lru 00000080:00200000:2.0:1787827204.838281:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000e09e5cd8 252: 1< 0 - - > 2< 1032192 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.838284:0:48445:0:(rw26.c:933:ll_write_end()) end page@0000000026c57f9f 00000080:00200000:2.0:1787827204.838285:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 253. 00000080:00200000:2.0:1787827204.838287:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 253 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.838301:0:48445:0:(rw26.c:875:ll_write_end()) pos 1036288, len 4096, copied 4096 00000080:00200000:2.0:1787827204.838306:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000eb5d8792[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 253] 00000080:00200000:2.0:1787827204.838309:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000001b133752 17ffffc0001001 3:0 ffff8ab3e0961910 253 lru 00000080:00200000:2.0:1787827204.838315:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000006865d8ab 253: 1< 0 - - > 2< 1036288 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.838317:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000eb5d8792 00000080:00200000:2.0:1787827204.838318:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 254. 00000080:00200000:2.0:1787827204.838320:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 254 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.838333:0:48445:0:(rw26.c:875:ll_write_end()) pos 1040384, len 4096, copied 4096 00000080:00200000:2.0:1787827204.838336:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000d14107a5[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 254] 00000080:00200000:2.0:1787827204.838339:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @000000000d2722da 17ffffc0001001 3:0 ffff8ab3e0961828 254 lru 00000080:00200000:2.0:1787827204.838345:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@00000000068f277a 254: 1< 0 - - > 2< 1040384 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.838348:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000d14107a5 00000080:00200000:2.0:1787827204.838349:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 255. 00000080:00200000:2.0:1787827204.838351:0:48445:0:(rw26.c:681:ll_write_begin()) Writing 255 of 0 to 4096 bytes 00000080:00200000:2.0:1787827204.838365:0:48445:0:(rw26.c:875:ll_write_end()) pos 1044480, len 4096, copied 4096 00000080:00200000:2.0:1787827204.838369:0:48445:0:(rw26.c:933:ll_write_end()) page@00000000286f44cc[3 00000000c2d04c9a 2 1 000000007cb5813e 0 0 255] 00000080:00200000:2.0:1787827204.838373:0:48445:0:(rw26.c:933:ll_write_end()) vmpage @00000000e6655880 17ffffc0001001 3:0 ffff8ab3e0961cb0 255 lru 00000080:00200000:2.0:1787827204.838378:0:48445:0:(rw26.c:933:ll_write_end()) osc-page@000000004dddfa86 255: 1< 0 - - > 2< 1044480 0 0 0x0 0x0 | 000000006a588f83 0000000066060182 > 3< 0 0 > 4< 0 0 8 2392064 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1787827204.838381:0:48445:0:(rw26.c:933:ll_write_end()) end page@00000000286f44cc 00000080:00200000:2.0:1787827204.838382:0:48445:0:(rw26.c:933:ll_write_end()) queued cl_page: 256. 00000080:00200000:2.0:1787827204.838387:0:48445:0:(vvp_io.c:1214:vvp_io_write_commit()) commit async pages: 256, from 0, to 4096 prio 0 00000008:00010000:2.0:1787827204.838428:0:48445:0:(osc_cache.c:858:osc_extent_find()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838456:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838468:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838478:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838488:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838498:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838507:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838518:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838528:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838538:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838548:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838560:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838569:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838578:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838588:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.838607:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.838615:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838625:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838635:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838650:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838659:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838668:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838678:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838688:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838699:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838708:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838718:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838728:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838738:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838747:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838757:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.838771:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.838791:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838806:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838816:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838825:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838836:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838851:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838860:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838870:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838879:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838909:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838920:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838931:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838941:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838952:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838963:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.838978:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.838985:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.838995:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839005:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839014:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839024:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839033:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839042:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839059:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839068:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839078:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839088:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839113:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839123:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839133:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839143:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.839157:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.839164:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839174:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839184:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839193:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839203:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839213:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839221:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839230:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839239:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839256:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839266:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839276:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839286:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839296:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839305:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.839319:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.839325:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839334:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839348:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839357:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839366:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839376:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839386:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839396:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839406:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839415:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839425:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839435:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839445:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839453:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839461:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.839474:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.839479:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839489:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839498:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839508:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839522:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839531:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839541:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839550:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839558:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839566:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839574:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839583:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839591:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839600:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839609:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.839621:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.839626:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839634:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839643:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839652:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839662:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839672:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839687:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839696:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839706:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839715:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839725:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839733:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839742:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839752:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839763:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.839777:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.839783:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839793:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839803:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839813:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839822:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839832:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839842:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839853:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839869:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839879:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839910:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839921:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839930:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839939:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839947:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.839962:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.839967:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839982:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.839992:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840002:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840011:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840020:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840029:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840040:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840050:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840060:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840071:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840080:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840090:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840112:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840122:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.840136:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.840142:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840152:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840162:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840178:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840188:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840197:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840206:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840216:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840226:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840234:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840243:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840254:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840265:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840274:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840284:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.840297:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.840303:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840312:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840323:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840332:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840342:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840358:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840368:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840379:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840389:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840399:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840410:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840420:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840431:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840442:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840452:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.840466:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.840472:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840482:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840492:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840502:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840512:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840522:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840532:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840548:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840558:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840569:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840578:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840588:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840598:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840609:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840619:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.840634:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.840641:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840652:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840663:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840674:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840684:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840694:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840705:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840716:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840727:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840744:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840756:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840766:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840777:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840788:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840799:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.840813:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.840819:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840829:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840845:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840854:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840865:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840875:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840905:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840918:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840928:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840939:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840950:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840961:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840970:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840980:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.840990:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.841005:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.841011:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841021:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841031:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841041:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841056:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841066:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841075:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841086:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841111:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841121:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841131:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841141:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841151:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841161:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841171:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.841186:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.841191:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841201:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841212:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841221:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841230:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841238:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841254:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841264:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841275:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841284:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841293:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841303:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841313:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841323:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827204.841332:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.841346:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 15, dirtied 15 00000008:00010000:2.0:1787827204.841351:0:48445:0:(osc_cache.c:2567:osc_queue_async_io()) ### extent: 0000000005e9ef40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827204.841361:0:48445:0:(vvp_io.c:1127:vvp_set_batch_dirty()) mapping ffff8ab3f11fedb8, count 1, dirtied 1 00000080:00200000:2.0:1787827204.841365:0:48445:0:(vvp_io.c:1237:vvp_io_write_commit()) Committed 256 pages 1048576 bytes, tot: 1048576 00000080:00200000:2.0:1787827204.841369:0:48445:0:(vvp_io.c:1421:vvp_io_write_start()) f2: write bytes 0, result: 1048576 00000020:00010000:2.0:1787827204.841386:0:48445:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8ab3c6a258a8 (ffff8ab3c6ae12d8/0) at cl_lock_release():245 00000020:00010000:2.0:1787827204.841390:0:48445:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3c6a258a8 (ffff8ab3c6ae12d8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827204.841394:0:48445:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8ab3d068eeb0 (ffff8ab3c6ae1ab8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1787827204.841402:0:48445:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827204.841416:0:48445:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fa1cd311/0x268e1a602cc25c8 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0da907 expref: -99 pid: 48445 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1787827204.841436:0:48445:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3c6a258a8 (ffff8ab3c6ae12d8/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827204.841439:0:48445:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8ab3d068eeb0 (ffff8ab3c6ae12d8/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827204.841452:0:48445:0:(vvp_io.c:533:vvp_io_advance()) advancing 1048576 bytes 00000080:00200000:2.0:1787827204.841460:0:48445:0:(file.c:2207:ll_file_io_generic()) Range unlock [0, 255] 00000080:00200000:2.0:1787827204.841470:0:48445:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x51:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827204.841474:0:48445:0:(file.c:2240:ll_file_io_generic()) f2: 2 io complete with rc: 0, result: 1048576, restart: 0 00000080:00200000:2.0:1787827204.841489:0:48445:0:(file.c:1775:ll_merge_attr_nolock()) [0x240001b71:0x51:0x0] updating i_size 1048576 i_blocks 0 00000080:00200000:2.0:1787827204.841496:0:48445:0:(file.c:2326:ll_file_io_generic()) iot: 2, result: 1048576 00000080:00200008:2.0:1787827204.841539:0:48445:0:(file.c:479:ll_file_release()) START file f2:[0x240001b71:0x51:0x0](ffff8ab3f11fec08), flags 100001 00000002:00100000:2.0:1787827204.841616:0:48445:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8ab3d0a27840 x1874671280572288/t17179869534(17179869534) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 576/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1787827204.841636:0:48445:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1787827204.841641:0:48445:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a24b40 x1874671280572672/t0(0) o35->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1787827204.841665:0:48445:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a24b40 pname:cluuid:pid:xid:nid:opc:job dd:27aedc06-3022-4c11-aafb-6bd27e69b836:48445:1874671280572672:192.168.203.124@tcp:35:dd.0 00000100:00100000:2.0:1787827204.841755:0:48445:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79139c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.843082:0:48445:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a24b40 pname:cluuid:pid:xid:nid:opc:job dd:27aedc06-3022-4c11-aafb-6bd27e69b836:48445:1874671280572672:192.168.203.124@tcp:35:dd.0 00000002:00100000:2.0:1787827204.843123:0:48445:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8ab3d0a27840 x1874671280572288/t17179869534(17179869534) o101->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 576/664 e 0 to 0 dl 1787827220 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787827204.849087:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.849127:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.849146:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.849159:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.849166:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.849172:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.849175:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.849182:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.849185:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.849192:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.849210:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.849216:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.849221:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.849224:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.849227:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.849236:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub0, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.849239:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.849242:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.849245:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub0, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.849248:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub0, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.849304:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.849314:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a25e00 x1874671280572928/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.849332:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a25e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280572928:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.849400:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79139c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.851031:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a25e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280572928:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.851063:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x52:0x0](ffff8ab3f11fba88) 00000080:00200000:2.0:1787827204.851113:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub0, unhashed 1 00000080:00200000:2.0:1787827204.851139:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.851147:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.851156:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.851164:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.851174:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.851178:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.851180:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.851183:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.851186:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.851192:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.851199:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.851202:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.851205:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.851207:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.851208:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.851215:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub1, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.851217:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.851220:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.851223:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub1, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.851225:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub1, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.851248:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.851254:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a24000 x1874671280573184/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.851265:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a24000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280573184:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.851302:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79139c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.852500:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a24000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280573184:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.852522:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x53:0x0](ffff8ab3f11fd348) 00000080:00200000:2.0:1787827204.852536:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub1, unhashed 1 00000080:00200000:2.0:1787827204.852554:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.852562:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.852570:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.852578:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.852584:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.852588:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.852590:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.852593:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.852596:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.852602:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.852609:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.852613:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.852615:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.852617:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.852619:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.852625:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub2, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.852628:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.852631:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.852634:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub2, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.852636:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub2, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.852656:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.852661:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a25a40 x1874671280573440/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.852680:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a25a40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280573440:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.852709:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79139c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.853786:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a25a40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280573440:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.853806:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x54:0x0](ffff8ab3f11ff448) 00000080:00200000:2.0:1787827204.853821:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub2, unhashed 1 00000080:00200000:2.0:1787827204.853842:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.853852:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.853862:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.853873:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.853882:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.853910:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.853913:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.853918:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.853921:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.853933:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.853942:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.853946:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.853951:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.853953:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.853955:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.853963:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub3, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.853965:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.853969:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.853971:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub3, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.853973:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub3, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.853994:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.854000:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a270c0 x1874671280573696/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.854011:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a270c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280573696:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.854044:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79139c0 going to sleep for 11 seconds 00000100:00100000:0.0:1787827204.855110:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a270c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280573696:192.168.203.124@tcp:36:createmany.0 00000080:00200000:0.0:1787827204.855141:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x55:0x0](ffff8ab3f123b248) 00000080:00200000:0.0:1787827204.855178:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub3, unhashed 1 00000080:00200000:0.0:1787827204.855209:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827204.855218:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827204.855228:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.855240:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.855247:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827204.855253:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827204.855256:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827204.855261:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:0.0:1787827204.855266:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827204.855275:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827204.855287:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.855292:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:0.0:1787827204.855297:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.855301:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:0.0:1787827204.855306:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:0.0:1787827204.855315:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub4, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:0.0:1787827204.855320:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.855325:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:0.0:1787827204.855328:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub4, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:0.0:1787827204.855331:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub4, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:0.0:1787827204.855356:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787827204.855362:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0aada40 x1874671280573952/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827204.855377:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aada40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280573952:192.168.203.124@tcp:36:createmany.0 00000100:00100000:0.0:1787827204.855414:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c6a4cc00 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.856451:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aada40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280573952:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.856469:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x56:0x0](ffff8ab3f11f80c8) 00000080:00200000:2.0:1787827204.856483:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub4, unhashed 1 00000080:00200000:2.0:1787827204.856508:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.856516:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.856523:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.856531:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.856537:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.856540:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.856542:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.856546:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.856548:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.856554:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.856561:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.856563:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.856566:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.856569:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.856572:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.856581:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub5, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.856584:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.856589:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.856592:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub5, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.856596:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub5, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.856624:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.856631:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a26d00 x1874671280574208/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.856647:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a26d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280574208:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.856680:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.857646:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a26d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280574208:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.857664:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x57:0x0](ffff8ab3f11fcb08) 00000080:00200000:2.0:1787827204.857677:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub5, unhashed 1 00000080:00200000:2.0:1787827204.857695:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.857712:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.857722:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.857732:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.857741:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.857747:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.857750:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.857756:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.857760:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.857767:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.857774:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.857777:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.857780:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.857783:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.857785:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.857793:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub6, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.857795:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.857798:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.857800:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub6, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.857803:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub6, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.857823:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.857828:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a243c0 x1874671280574464/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.857840:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a243c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280574464:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.857870:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.858856:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a243c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280574464:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.858872:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x58:0x0](ffff8ab3f11fe3c8) 00000080:00200000:2.0:1787827204.858913:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub6, unhashed 1 00000080:00200000:2.0:1787827204.858932:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.858938:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.858945:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.858952:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.858957:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.858960:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.858963:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.858966:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.858968:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.858974:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.858981:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.858983:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.858985:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.858987:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.858989:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.858994:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub7, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.858996:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.858999:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.859001:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub7, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.859004:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub7, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.859022:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.859027:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a25680 x1874671280574720/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.859038:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a25680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280574720:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.859067:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.860160:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a25680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280574720:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.860178:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x59:0x0](ffff8ab3f11faa08) 00000080:00200000:2.0:1787827204.860191:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub7, unhashed 1 00000080:00200000:2.0:1787827204.860208:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.860215:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.860222:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.860229:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.860234:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.860237:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.860239:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.860243:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.860245:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.860260:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.860267:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.860269:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.860272:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.860274:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.860276:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.860283:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub8, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.860286:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.860289:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.860291:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub8, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.860294:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub8, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.860314:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.860319:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a27480 x1874671280574976/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.860334:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a27480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280574976:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.860365:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.861424:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a27480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280574976:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.861440:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x5a:0x0](ffff8ab3f11f9988) 00000080:00200000:2.0:1787827204.861453:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub8, unhashed 1 00000080:00200000:2.0:1787827204.861471:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.861477:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.861484:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.861502:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.861510:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.861515:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.861518:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.861523:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.861527:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.861535:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.861547:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.861551:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.861554:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.861556:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.861558:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.861566:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub9, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.861569:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.861574:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.861578:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub9, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.861581:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub9, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.861609:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.861617:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a24780 x1874671280575232/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.861630:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a24780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280575232:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.861660:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.862632:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a24780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280575232:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.862669:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x5b:0x0](ffff8ab3f13fdb88) 00000080:00200000:2.0:1787827204.862683:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub9, unhashed 1 00000080:00200000:2.0:1787827204.862700:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.862707:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.862716:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.862739:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.862747:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.862753:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.862756:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.862761:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.862765:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.862773:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.862783:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.862786:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.862789:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.862791:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.862793:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.862800:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub10, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.862802:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.862805:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.862807:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub10, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.862809:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub10, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.862830:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.862834:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a252c0 x1874671280575488/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.862845:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a252c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280575488:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.862875:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.863914:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a252c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280575488:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.863931:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x5c:0x0](ffff8ab3f13f8908) 00000080:00200000:2.0:1787827204.863944:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub10, unhashed 1 00000080:00200000:2.0:1787827204.863969:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.863976:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.863983:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.863991:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.863996:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.863999:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.864001:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.864005:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.864007:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.864013:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.864019:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.864022:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.864024:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.864026:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.864028:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.864034:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub11, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.864036:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.864039:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.864041:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub11, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.864044:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub11, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.864062:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.864067:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a26940 x1874671280575744/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.864082:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a26940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280575744:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.864126:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.865068:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a26940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280575744:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.865085:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x5d:0x0](ffff8ab3f13fb248) 00000080:00200000:2.0:1787827204.865114:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub11, unhashed 1 00000080:00200000:2.0:1787827204.865132:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.865140:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.865147:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.865155:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.865160:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.865164:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.865166:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.865169:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.865172:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.865178:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.865185:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.865188:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.865190:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.865193:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.865195:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.865201:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub12, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.865203:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.865206:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.865208:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub12, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.865211:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub12, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.865229:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.865235:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a261c0 x1874671280576000/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.865254:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a261c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280576000:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.865284:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.866250:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a261c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280576000:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.866268:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x5e:0x0](ffff8ab3f13fa1c8) 00000080:00200000:2.0:1787827204.866283:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub12, unhashed 1 00000080:00200000:2.0:1787827204.866306:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.866316:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.866325:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.866336:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.866345:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.866349:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.866351:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.866357:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.866359:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.866365:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.866371:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.866374:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.866377:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.866380:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.866382:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.866389:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub13, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.866392:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.866395:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.866397:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub13, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.866399:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub13, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.866420:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.866426:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e089d680 x1874671280576256/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.866437:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e089d680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280576256:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.866474:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.867480:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e089d680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280576256:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.867499:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x5f:0x0](ffff8ab3f13fc2c8) 00000080:00200000:2.0:1787827204.867536:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub13, unhashed 1 00000080:00200000:2.0:1787827204.867554:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.867563:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.867570:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.867579:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.867595:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.867609:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.867612:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.867617:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.867621:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.867629:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.867640:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.867645:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.867650:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.867653:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.867655:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.867663:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub14, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.867667:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.867671:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.867674:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub14, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.867677:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub14, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.867704:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.867711:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e089f480 x1874671280576512/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.867722:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e089f480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280576512:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.867756:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.868918:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e089f480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280576512:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.868937:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x60:0x0](ffff8ab3f13fec08) 00000080:00200000:2.0:1787827204.868949:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub14, unhashed 1 00000080:00200000:2.0:1787827204.868966:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.868973:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.868980:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.868987:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.868993:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.868996:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.868998:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.869002:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.869004:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.869010:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.869017:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.869019:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.869022:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.869024:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.869026:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.869032:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub15, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.869034:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.869037:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.869039:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub15, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.869042:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub15, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.869060:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.869065:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e089c000 x1874671280576768/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.869084:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e089c000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280576768:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.869138:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.870337:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e089c000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280576768:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.870355:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x61:0x0](ffff8ab3f13fba88) 00000080:00200000:2.0:1787827204.870369:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub15, unhashed 1 00000080:00200000:2.0:1787827204.870387:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.870395:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.870402:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.870410:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.870415:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.870418:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.870420:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.870424:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.870427:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.870433:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.870439:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.870443:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.870445:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.870447:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.870449:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.870456:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub16, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.870458:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.870461:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.870463:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub16, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.870466:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub16, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.870492:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.870512:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e089de00 x1874671280577024/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.870538:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e089de00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280577024:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.870572:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.871686:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e089de00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280577024:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.871706:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x62:0x0](ffff8ab3f13fd348) 00000080:00200000:2.0:1787827204.871724:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub16, unhashed 1 00000080:00200000:2.0:1787827204.871748:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.871759:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.871769:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.871777:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.871782:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.871786:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.871788:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.871791:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.871794:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.871800:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.871807:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.871809:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.871812:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.871814:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.871815:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.871822:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub17, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.871824:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.871827:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.871830:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub17, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.871832:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub17, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.871852:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.871857:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e089da40 x1874671280577280/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.871868:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e089da40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280577280:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.871919:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.873144:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e089da40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280577280:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.873178:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x63:0x0](ffff8ab3f13ff448) 00000080:00200000:2.0:1787827204.873196:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub17, unhashed 1 00000080:00200000:2.0:1787827204.873216:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.873224:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.873245:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.873257:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.873267:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.873272:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.873276:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.873281:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.873284:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.873290:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.873298:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.873301:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.873305:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.873307:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.873309:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.873317:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub18, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.873320:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.873323:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.873325:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub18, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.873327:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub18, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.873349:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.873354:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e089c3c0 x1874671280577536/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.873366:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e089c3c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280577536:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.873397:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.874378:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e089c3c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280577536:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.874396:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x64:0x0](ffff8ab3f13f80c8) 00000080:00200000:2.0:1787827204.874408:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub18, unhashed 1 00000080:00200000:2.0:1787827204.874425:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.874433:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.874441:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.874451:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.874492:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.874496:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.874500:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.874504:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.874507:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.874514:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.874523:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.874527:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.874530:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.874532:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.874535:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.874542:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub19, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.874544:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.874547:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.874550:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub19, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.874552:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub19, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.874572:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.874578:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e089fc00 x1874671280577792/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.874588:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e089fc00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280577792:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.874621:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.875681:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e089fc00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280577792:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.875701:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x65:0x0](ffff8ab3f13fcb08) 00000080:00200000:2.0:1787827204.875715:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub19, unhashed 1 00000080:00200000:2.0:1787827204.875737:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.875748:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.875757:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.875769:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.875777:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.875782:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.875786:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.875789:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.875792:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.875798:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.875805:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.875808:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.875811:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.875813:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.875815:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.875821:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub20, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.875823:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.875827:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.875829:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub20, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.875832:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub20, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.875853:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.875858:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e089d2c0 x1874671280578048/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.875869:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e089d2c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280578048:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.875917:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.876988:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e089d2c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280578048:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.877004:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x66:0x0](ffff8ab3f13fe3c8) 00000080:00200000:2.0:1787827204.877016:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub20, unhashed 1 00000080:00200000:2.0:1787827204.877034:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.877040:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.877047:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.877054:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.877059:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.877063:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.877064:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.877068:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.877070:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.877076:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.877082:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.877085:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.877088:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.877090:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.877092:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.877112:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub21, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.877115:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.877118:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.877120:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub21, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.877122:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub21, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.877144:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.877148:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e089e1c0 x1874671280578304/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.877160:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e089e1c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280578304:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.877190:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.878092:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e089e1c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280578304:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.878122:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x67:0x0](ffff8ab3f13faa08) 00000080:00200000:2.0:1787827204.878135:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub21, unhashed 1 00000080:00200000:2.0:1787827204.878152:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.878159:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.878165:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.878172:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.878177:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.878180:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.878182:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.878186:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.878188:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.878194:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.878200:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.878203:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.878205:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.878206:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.878208:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.878214:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub22, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.878216:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.878219:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.878221:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub22, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.878224:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub22, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.878250:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.878257:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e089cf00 x1874671280578560/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.878274:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e089cf00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280578560:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.878308:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.879289:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e089cf00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280578560:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.879309:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x68:0x0](ffff8ab3f13f9988) 00000080:00200000:2.0:1787827204.879325:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub22, unhashed 1 00000080:00200000:2.0:1787827204.879347:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.879357:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.879366:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.879377:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.879384:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.879389:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.879392:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.879396:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.879399:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.879408:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.879418:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.879423:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.879427:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.879430:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.879433:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.879442:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub23, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.879445:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.879450:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.879453:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub23, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.879456:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub23, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.879483:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.879491:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e089e940 x1874671280578816/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.879522:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e089e940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280578816:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.879583:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.880629:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e089e940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280578816:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.880652:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x69:0x0](ffff8ab3f13f9148) 00000080:00200000:2.0:1787827204.880668:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub23, unhashed 1 00000080:00200000:2.0:1787827204.880690:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.880699:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.880709:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.880719:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.880726:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.880731:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.880734:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.880739:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.880742:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.880752:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.880763:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.880768:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.880772:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.880775:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.880778:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.880787:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub24, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.880790:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.880795:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.880799:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub24, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.880803:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub24, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.880833:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.880840:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e08a7840 x1874671280579072/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.880856:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e08a7840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280579072:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.880913:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.881916:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e08a7840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280579072:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.881961:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x6a:0x0](ffff8ab3f12bd348) 00000080:00200000:2.0:1787827204.881979:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub24, unhashed 1 00000080:00200000:2.0:1787827204.882001:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.882010:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.882020:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.882031:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.882039:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.882043:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.882047:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.882051:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.882055:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.882065:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.882075:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.882080:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.882084:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.882087:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.882090:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.882115:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub25, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.882119:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.882123:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.882127:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub25, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.882130:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub25, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.882158:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.882166:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e08a6d00 x1874671280579328/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.882181:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e08a6d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280579328:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.882223:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.883191:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e08a6d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280579328:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.883214:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x6b:0x0](ffff8ab3f12bf448) 00000080:00200000:2.0:1787827204.883232:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub25, unhashed 1 00000080:00200000:2.0:1787827204.883254:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.883265:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.883275:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.883286:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.883294:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.883299:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.883302:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.883307:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.883311:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.883321:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.883331:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.883337:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.883341:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.883345:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.883348:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.883357:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub26, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.883361:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.883365:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.883369:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub26, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.883372:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub26, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.883401:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.883409:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e08a5e00 x1874671280579584/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.883424:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e08a5e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280579584:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.883466:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.884445:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e08a5e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280579584:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.884462:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x6c:0x0](ffff8ab3f12b80c8) 00000080:00200000:2.0:1787827204.884476:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub26, unhashed 1 00000080:00200000:2.0:1787827204.884497:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.884506:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.884515:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.884526:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.884533:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.884537:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.884540:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.884544:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.884547:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.884555:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.884565:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.884569:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.884573:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.884577:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.884580:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.884589:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub27, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.884592:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.884596:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.884600:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub27, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.884603:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub27, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.884627:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.884634:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e08a7480 x1874671280579840/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.884648:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e08a7480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280579840:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.884685:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.885649:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e08a7480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280579840:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.885666:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x6d:0x0](ffff8ab3f12bcb08) 00000080:00200000:2.0:1787827204.885679:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub27, unhashed 1 00000080:00200000:2.0:1787827204.885695:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.885702:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.885709:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.885716:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.885720:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.885723:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.885726:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.885729:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.885732:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.885737:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.885744:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.885747:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.885749:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.885751:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.885753:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.885758:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub28, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.885761:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.885763:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.885766:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub28, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.885768:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub28, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.885786:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.885791:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0b62d00 x1874671280580096/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.885802:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0b62d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280580096:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.885831:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.886834:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0b62d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280580096:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.886851:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x6e:0x0](ffff8ab3f12be3c8) 00000080:00200000:2.0:1787827204.886863:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub28, unhashed 1 00000080:00200000:2.0:1787827204.886880:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.886905:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.886912:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.886919:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.886924:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.886927:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.886929:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.886932:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.886935:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.886941:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.886948:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.886951:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.886953:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.886955:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.886956:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.886964:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub29, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.886966:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.886968:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.886971:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub29, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.886973:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub29, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.886992:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.886997:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0b60000 x1874671280580352/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.887008:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0b60000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280580352:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.887038:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.888133:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0b60000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280580352:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.888151:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x6f:0x0](ffff8ab3f12baa08) 00000080:00200000:2.0:1787827204.888164:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub29, unhashed 1 00000080:00200000:2.0:1787827204.888183:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.888190:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.888198:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.888206:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.888211:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.888215:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.888217:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.888220:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.888223:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.888229:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.888235:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.888239:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.888242:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.888244:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.888246:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.888253:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub30, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.888256:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.888258:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.888261:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub30, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.888263:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub30, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.888283:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.888288:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0b63840 x1874671280580608/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.888300:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0b63840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280580608:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.888332:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.889400:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0b63840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280580608:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.889428:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x70:0x0](ffff8ab3f12b9988) 00000080:00200000:2.0:1787827204.889449:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub30, unhashed 1 00000080:00200000:2.0:1787827204.889477:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.889487:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.889499:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.889510:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.889519:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.889524:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.889527:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.889533:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.889537:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.889546:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.889558:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.889564:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.889568:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.889571:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.889575:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.889585:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub31, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.889588:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.889593:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.889596:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub31, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.889600:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub31, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.889631:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.889639:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0b62940 x1874671280580864/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.889657:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0b62940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280580864:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.889720:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.890750:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0b62940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280580864:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.890774:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x71:0x0](ffff8ab3f12b9148) 00000080:00200000:2.0:1787827204.890790:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub31, unhashed 1 00000080:00200000:2.0:1787827204.890812:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.890822:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.890832:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.890844:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.890851:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.890856:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.890860:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.890864:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.890868:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.890877:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.890933:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.890940:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.890944:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.890948:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.890951:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.890961:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub32, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.890965:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.890970:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.890974:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub32, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.890978:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub32, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.891007:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.891016:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0b60b40 x1874671280581120/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.891032:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0b60b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280581120:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.891074:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.892241:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0b60b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280581120:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.892262:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x72:0x0](ffff8ab3f12bdb88) 00000080:00200000:2.0:1787827204.892280:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub32, unhashed 1 00000080:00200000:2.0:1787827204.892302:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.892312:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.892322:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.892333:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.892341:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.892346:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.892349:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.892354:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.892358:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.892367:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.892377:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.892382:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.892387:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.892390:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.892393:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.892403:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub33, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.892406:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.892410:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.892414:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub33, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.892417:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub33, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.892444:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.892451:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0b60780 x1874671280581376/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.892468:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0b60780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280581376:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.892507:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.893446:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0b60780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280581376:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.893469:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x73:0x0](ffff8ab3f12b8908) 00000080:00200000:2.0:1787827204.893500:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub33, unhashed 1 00000080:00200000:2.0:1787827204.893518:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.893525:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.893532:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.893540:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.893545:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.893548:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.893550:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.893554:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.893556:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.893562:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.893568:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.893571:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.893574:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.893575:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.893577:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.893583:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub34, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.893586:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.893588:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.893591:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub34, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.893593:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub34, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.893611:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.893617:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0b630c0 x1874671280581632/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.893627:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0b630c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280581632:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.893660:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.894612:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0b630c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280581632:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.894627:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x74:0x0](ffff8ab3f12bb248) 00000080:00200000:2.0:1787827204.894639:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub34, unhashed 1 00000080:00200000:2.0:1787827204.894656:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.894663:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.894670:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.894676:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.894681:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.894684:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.894686:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.894689:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.894692:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.894698:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.894713:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.894715:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.894718:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.894720:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.894722:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.894728:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub35, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.894730:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.894733:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.894736:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub35, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.894738:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub35, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.894756:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.894761:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0b63480 x1874671280581888/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.894771:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0b63480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280581888:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.894801:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.895702:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0b63480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280581888:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.895719:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x75:0x0](ffff8ab3f12ba1c8) 00000080:00200000:2.0:1787827204.895732:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub35, unhashed 1 00000080:00200000:2.0:1787827204.895748:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.895754:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.895760:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.895767:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.895772:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.895775:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.895777:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.895780:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.895782:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.895788:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.895794:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.895797:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.895800:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.895802:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.895803:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.895809:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub36, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.895811:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.895814:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.895816:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub36, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.895818:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub36, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.895835:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.895840:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0b60f00 x1874671280582144/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.895850:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0b60f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280582144:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.895878:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.896958:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0b60f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280582144:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.896973:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x76:0x0](ffff8ab3f12bc2c8) 00000080:00200000:2.0:1787827204.896986:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub36, unhashed 1 00000080:00200000:2.0:1787827204.897002:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.897009:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.897015:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.897022:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.897027:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.897029:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.897032:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.897036:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.897038:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.897044:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.897050:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.897052:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.897055:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.897056:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.897058:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.897064:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub37, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.897066:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.897069:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.897071:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub37, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.897073:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub37, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.897090:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.897112:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0b603c0 x1874671280582400/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.897123:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0b603c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280582400:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.897152:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.898182:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0b603c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280582400:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.898199:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x77:0x0](ffff8ab3f12bec08) 00000080:00200000:2.0:1787827204.898211:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub37, unhashed 1 00000080:00200000:2.0:1787827204.898227:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.898234:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.898240:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.898247:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.898252:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.898255:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.898257:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.898260:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.898262:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.898268:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.898274:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.898277:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.898279:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.898281:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.898283:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.898288:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub38, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.898291:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.898293:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.898296:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub38, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.898298:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub38, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.898316:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.898321:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e0f04f00 x1874671280582656/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.898331:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0f04f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280582656:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.898360:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.899354:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0f04f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280582656:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.899375:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x78:0x0](ffff8ab3f12bba88) 00000080:00200000:2.0:1787827204.899392:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub38, unhashed 1 00000080:00200000:2.0:1787827204.899409:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.899418:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.899427:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.899436:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.899444:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.899449:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.899452:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.899456:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.899460:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.899468:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.899477:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.899482:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.899487:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.899490:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.899493:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.899510:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub39, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.899514:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.899518:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.899522:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub39, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.899525:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub39, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.899551:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.899558:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3e0f05e00 x1874671280582912/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.899574:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0f05e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280582912:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.899632:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.900616:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0f05e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280582912:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.900648:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x79:0x0](ffff8ab3f12c2a08) 00000080:00200000:2.0:1787827204.900662:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub39, unhashed 1 00000080:00200000:2.0:1787827204.900678:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.900685:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.900691:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.900698:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.900703:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.900706:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.900708:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.900712:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.900714:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.900720:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.900726:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.900729:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.900731:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.900732:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.900734:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.900740:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub40, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.900742:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.900745:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.900747:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub40, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.900749:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub40, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.900777:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.900782:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70ae1c0 x1874671280583168/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.900792:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70ae1c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280583168:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.900822:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.901848:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70ae1c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280583168:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.901865:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x7a:0x0](ffff8ab3f12c1988) 00000080:00200000:2.0:1787827204.901876:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub40, unhashed 1 00000080:00200000:2.0:1787827204.901927:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.901934:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.901941:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.901948:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.901954:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.901958:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.901960:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.901964:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.901967:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.901974:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.901980:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.901983:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.901985:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.901987:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.901989:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.901994:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub41, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.901996:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.901999:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.902002:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub41, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.902004:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub41, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.902021:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.902026:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70ad680 x1874671280583424/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.902035:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70ad680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280583424:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.902063:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.903089:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70ad680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280583424:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.903115:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x7b:0x0](ffff8ab3f12c1148) 00000080:00200000:2.0:1787827204.903128:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub41, unhashed 1 00000080:00200000:2.0:1787827204.903144:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.903150:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.903157:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.903163:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.903168:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.903171:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.903173:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.903176:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.903178:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.903184:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.903190:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.903194:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.903198:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.903201:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.903205:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.903213:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub42, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.903217:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.903222:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.903225:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub42, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.903228:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub42, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.903252:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.903256:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70afc00 x1874671280583680/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.903266:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70afc00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280583680:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.903295:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.904293:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70afc00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280583680:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.904315:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x7c:0x0](ffff8ab3f12c5b88) 00000080:00200000:2.0:1787827204.904333:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub42, unhashed 1 00000080:00200000:2.0:1787827204.904354:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.904363:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.904371:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.904379:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.904384:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.904388:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.904390:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.904393:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.904396:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.904401:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.904409:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.904412:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.904416:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.904419:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.904422:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.904431:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub43, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.904434:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.904438:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.904441:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub43, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.904445:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub43, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.904471:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.904478:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70ada40 x1874671280583936/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.904503:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70ada40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280583936:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.904538:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.905514:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70ada40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280583936:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.905536:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x7d:0x0](ffff8ab3f12c0908) 00000080:00200000:2.0:1787827204.905550:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub43, unhashed 1 00000080:00200000:2.0:1787827204.905571:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.905580:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.905589:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.905600:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.905607:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.905611:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.905614:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.905619:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.905623:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.905632:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.905642:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.905646:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.905651:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.905654:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.905657:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.905666:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub44, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.905669:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.905673:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.905677:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub44, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.905681:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub44, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.905704:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.905711:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70ad2c0 x1874671280584192/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.905726:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70ad2c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280584192:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.905760:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.906833:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70ad2c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280584192:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.906852:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x7e:0x0](ffff8ab3f12c3248) 00000080:00200000:2.0:1787827204.906865:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub44, unhashed 1 00000080:00200000:2.0:1787827204.906919:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.906928:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.906936:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.906944:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.906950:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.906953:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.906955:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.906959:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.906961:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.906967:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.906974:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.906977:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.906980:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.906982:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.906984:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.906992:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub45, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.906994:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.906997:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.907000:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub45, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.907002:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub45, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.907023:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.907028:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70ac3c0 x1874671280584448/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.907039:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70ac3c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280584448:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.907074:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.908028:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70ac3c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280584448:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.908048:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x7f:0x0](ffff8ab3f12c21c8) 00000080:00200000:2.0:1787827204.908062:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub45, unhashed 1 00000080:00200000:2.0:1787827204.908081:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.908089:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.908112:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.908121:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.908127:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.908130:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.908132:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.908136:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.908139:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.908145:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.908152:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.908155:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.908158:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.908159:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.908161:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.908168:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub46, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.908170:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.908173:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.908176:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub46, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.908178:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub46, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.908200:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.908208:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70ae580 x1874671280584704/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.908224:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70ae580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280584704:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.908263:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.909158:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70ae580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280584704:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.909176:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x80:0x0](ffff8ab3f12c42c8) 00000080:00200000:2.0:1787827204.909189:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub46, unhashed 1 00000080:00200000:2.0:1787827204.909213:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.909225:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.909235:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.909247:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.909256:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.909261:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.909263:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.909266:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.909269:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.909275:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.909282:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.909285:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.909288:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.909290:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.909292:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.909299:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub47, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.909301:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.909304:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.909307:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub47, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.909309:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub47, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.909329:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.909335:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70ac780 x1874671280584960/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.909345:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70ac780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280584960:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.909398:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.910321:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70ac780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280584960:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.910338:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x81:0x0](ffff8ab3f12c6c08) 00000080:00200000:2.0:1787827204.910350:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub47, unhashed 1 00000080:00200000:2.0:1787827204.910367:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.910374:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.910381:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.910388:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.910393:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.910396:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.910398:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.910401:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.910403:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.910409:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.910415:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.910418:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.910421:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.910423:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.910424:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.910431:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub48, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.910433:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.910436:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.910439:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub48, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.910441:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub48, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.910458:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.910463:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70aed00 x1874671280585216/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.910474:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70aed00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280585216:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.910509:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.911424:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70aed00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280585216:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.911446:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x82:0x0](ffff8ab3f12c3a88) 00000080:00200000:2.0:1787827204.911462:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub48, unhashed 1 00000080:00200000:2.0:1787827204.911485:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.911494:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.911502:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.911512:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.911520:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.911525:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.911529:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.911534:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.911538:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.911547:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.911557:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.911561:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.911565:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.911568:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.911571:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.911579:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub49, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.911582:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.911586:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.911590:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub49, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.911593:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub49, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.911618:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.911625:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70acf00 x1874671280585472/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.911639:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70acf00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280585472:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.911676:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.912624:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70acf00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280585472:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.912641:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x83:0x0](ffff8ab3f12c5348) 00000080:00200000:2.0:1787827204.912652:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub49, unhashed 1 00000080:00200000:2.0:1787827204.912669:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.912676:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.912683:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.912689:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.912695:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.912697:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.912699:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.912702:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.912705:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.912711:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.912717:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.912720:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.912722:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.912723:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.912725:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.912731:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub50, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.912733:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.912736:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.912738:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub50, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.912741:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub50, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.912758:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.912763:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70af840 x1874671280585728/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.912773:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70af840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280585728:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.912802:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.913687:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70af840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280585728:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.913704:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x84:0x0](ffff8ab3f12c7448) 00000080:00200000:2.0:1787827204.913717:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub50, unhashed 1 00000080:00200000:2.0:1787827204.913737:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.913745:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.913752:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.913760:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.913766:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.913769:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.913771:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.913775:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.913777:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.913783:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.913790:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.913793:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.913796:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.913798:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.913800:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.913806:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub51, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.913808:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.913811:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.913813:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub51, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.913815:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub51, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.913834:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.913839:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70af480 x1874671280585984/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.913850:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70af480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280585984:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.913902:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.914951:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70af480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280585984:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.914969:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x85:0x0](ffff8ab3f12c00c8) 00000080:00200000:2.0:1787827204.914982:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub51, unhashed 1 00000080:00200000:2.0:1787827204.914999:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.915006:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.915013:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.915019:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.915024:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.915027:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.915029:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.915032:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.915035:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.915040:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.915047:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.915049:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.915052:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.915054:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.915055:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.915063:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub52, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.915065:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.915068:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.915070:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub52, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.915072:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub52, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.915090:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.915111:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70af0c0 x1874671280586240/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.915123:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70af0c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280586240:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.915153:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.916131:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70af0c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280586240:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.916148:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x86:0x0](ffff8ab3f12c4b08) 00000080:00200000:2.0:1787827204.916160:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub52, unhashed 1 00000080:00200000:2.0:1787827204.916177:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.916184:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.916190:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.916199:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.916207:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.916212:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.916216:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.916220:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.916224:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.916233:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.916244:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.916248:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.916252:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.916254:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.916256:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.916263:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub53, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.916265:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.916268:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.916270:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub53, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.916273:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub53, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.916292:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.916298:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70ade00 x1874671280586496/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.916308:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70ade00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280586496:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.916338:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.917325:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70ade00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280586496:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.917347:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x87:0x0](ffff8ab3f12c63c8) 00000080:00200000:2.0:1787827204.917379:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub53, unhashed 1 00000080:00200000:2.0:1787827204.917400:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.917408:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.917418:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.917429:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.917436:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.917440:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.917454:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.917459:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.917463:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.917472:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.917482:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.917486:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.917491:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.917494:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.917497:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.917506:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub54, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.917509:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.917513:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.917517:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub54, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.917520:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub54, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.917546:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.917554:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70ae940 x1874671280586752/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.917570:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70ae940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280586752:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.917609:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.918558:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70ae940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280586752:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.918591:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x88:0x0](ffff8ab3f12e0908) 00000080:00200000:2.0:1787827204.918605:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub54, unhashed 1 00000080:00200000:2.0:1787827204.918624:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.918632:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.918639:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.918647:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.918653:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.918656:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.918658:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.918662:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.918665:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.918671:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.918677:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.918680:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.918682:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.918684:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.918686:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.918692:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub55, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.918694:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.918697:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.918700:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub55, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.918702:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub55, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.918721:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.918727:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70acb40 x1874671280587008/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.918738:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70acb40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280587008:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.918782:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.919875:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70acb40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280587008:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.919916:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x89:0x0](ffff8ab3f12e3248) 00000080:00200000:2.0:1787827204.919930:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub55, unhashed 1 00000080:00200000:2.0:1787827204.919949:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.919957:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.919964:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.919972:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.919978:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.919981:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.919983:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.919987:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.919989:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.919995:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.920002:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.920005:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.920007:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.920009:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.920011:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.920017:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub56, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.920019:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.920022:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.920025:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub56, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.920028:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub56, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.920047:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.920053:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3c70ac000 x1874671280587264/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.920063:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3c70ac000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280587264:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.920111:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.921205:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3c70ac000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280587264:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.921224:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x8a:0x0](ffff8ab3f12e21c8) 00000080:00200000:2.0:1787827204.921238:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub56, unhashed 1 00000080:00200000:2.0:1787827204.921257:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.921264:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.921272:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.921280:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.921286:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.921289:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.921291:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.921294:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.921297:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.921303:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.921309:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.921312:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.921315:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.921317:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.921319:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.921326:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub57, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.921329:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.921332:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.921334:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub57, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.921336:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub57, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.921367:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.921373:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0892940 x1874671280587520/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.921385:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0892940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280587520:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.921418:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.922355:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0892940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280587520:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.922371:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x8b:0x0](ffff8ab3f12e42c8) 00000080:00200000:2.0:1787827204.922382:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub57, unhashed 1 00000080:00200000:2.0:1787827204.922399:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.922405:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.922412:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.922419:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.922424:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.922426:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.922429:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.922432:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.922435:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.922449:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.922456:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.922459:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.922462:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.922464:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.922465:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.922472:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub58, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.922474:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.922477:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.922479:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub58, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.922481:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub58, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.922499:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.922504:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0890b40 x1874671280587776/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.922514:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0890b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280587776:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.922543:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.923661:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0890b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280587776:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.923680:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x8c:0x0](ffff8ab3f12e6c08) 00000080:00200000:2.0:1787827204.923693:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub58, unhashed 1 00000080:00200000:2.0:1787827204.923712:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.923720:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.923728:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.923736:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.923742:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.923745:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.923747:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.923751:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.923753:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.923759:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.923766:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.923769:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.923771:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.923773:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.923775:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.923781:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub59, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.923783:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.923786:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.923789:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub59, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.923791:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub59, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.923810:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.923816:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0890000 x1874671280588032/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.923826:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0890000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280588032:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.923860:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.924946:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0890000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280588032:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.924964:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x8d:0x0](ffff8ab3f12e3a88) 00000080:00200000:2.0:1787827204.924976:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub59, unhashed 1 00000080:00200000:2.0:1787827204.924993:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.925000:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.925007:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.925015:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.925020:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.925023:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.925025:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.925028:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.925030:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.925036:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.925043:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.925045:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.925048:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.925050:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.925052:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.925058:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub60, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.925060:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.925063:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.925065:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub60, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.925067:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub60, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.925111:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.925117:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d08921c0 x1874671280588288/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.925128:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08921c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280588288:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.925160:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.926189:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d08921c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280588288:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.926205:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x8e:0x0](ffff8ab3f12e5348) 00000080:00200000:2.0:1787827204.926217:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub60, unhashed 1 00000080:00200000:2.0:1787827204.926234:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.926240:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.926247:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.926254:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.926258:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.926261:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.926264:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.926267:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.926270:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.926275:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.926282:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.926284:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.926286:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.926288:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.926290:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.926296:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub61, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.926299:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.926301:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.926304:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub61, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.926306:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub61, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.926324:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.926328:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0891680 x1874671280588544/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.926339:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0891680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280588544:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.926368:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.927414:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0891680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280588544:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.927431:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x8f:0x0](ffff8ab3f12e7448) 00000080:00200000:2.0:1787827204.927452:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub61, unhashed 1 00000080:00200000:2.0:1787827204.927468:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.927475:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.927482:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.927495:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.927501:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.927504:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.927506:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.927510:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.927514:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.927523:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.927533:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.927538:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.927542:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.927545:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.927548:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.927557:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub62, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.927560:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.927565:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.927569:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub62, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.927571:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub62, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.927592:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.927597:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0893c00 x1874671280588800/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.927607:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0893c00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280588800:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.927637:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.928705:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0893c00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280588800:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.928721:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x90:0x0](ffff8ab3f12e00c8) 00000080:00200000:2.0:1787827204.928733:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub62, unhashed 1 00000080:00200000:2.0:1787827204.928750:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.928756:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.928763:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.928769:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.928774:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.928778:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.928779:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.928782:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.928784:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.928790:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.928797:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.928799:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.928803:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.928805:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.928807:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.928812:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub63, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.928814:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.928817:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.928819:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub63, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.928821:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub63, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.928841:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.928848:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0891a40 x1874671280589056/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.928863:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0891a40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280589056:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.928934:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.929953:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0891a40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280589056:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.929970:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x91:0x0](ffff8ab3f12e4b08) 00000080:00200000:2.0:1787827204.929982:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub63, unhashed 1 00000080:00200000:2.0:1787827204.929998:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.930005:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.930011:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.930018:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.930022:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.930025:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.930027:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.930031:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.930034:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.930039:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.930046:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.930049:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.930051:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.930052:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.930054:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.930060:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub64, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.930062:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.930065:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.930067:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub64, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.930069:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub64, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.930086:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.930091:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d08912c0 x1874671280589312/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.930118:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08912c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280589312:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.930147:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.931126:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d08912c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280589312:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.931141:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x92:0x0](ffff8ab3f12e63c8) 00000080:00200000:2.0:1787827204.931152:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub64, unhashed 1 00000080:00200000:2.0:1787827204.931168:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.931175:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.931181:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.931188:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.931193:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.931196:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.931198:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.931201:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.931203:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.931210:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.931216:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.931218:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.931220:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.931222:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.931223:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.931229:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub65, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.931231:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.931234:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.931236:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub65, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.931238:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub65, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.931255:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.931260:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d08903c0 x1874671280589568/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.931270:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08903c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280589568:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.931299:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.932287:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d08903c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280589568:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.932305:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x93:0x0](ffff8ab3f12e2a08) 00000080:00200000:2.0:1787827204.932318:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub65, unhashed 1 00000080:00200000:2.0:1787827204.932337:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.932344:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.932351:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.932359:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.932364:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.932367:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.932369:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.932373:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.932375:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.932381:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.932388:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.932391:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.932394:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.932396:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.932398:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.932404:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub66, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.932406:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.932409:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.932412:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub66, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.932414:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub66, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.932442:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.932448:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0892580 x1874671280589824/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.932459:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0892580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280589824:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.932490:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.933416:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0892580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280589824:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.933434:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x94:0x0](ffff8ab3f12e1988) 00000080:00200000:2.0:1787827204.933445:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub66, unhashed 1 00000080:00200000:2.0:1787827204.933461:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.933468:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.933474:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.933481:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.933486:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.933489:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.933491:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.933494:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.933497:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.933503:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.933514:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.933518:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.933522:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.933525:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.933528:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.933536:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub67, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.933540:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.933544:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.933548:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub67, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.933552:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub67, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.933575:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.933580:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0890780 x1874671280590080/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.933589:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0890780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280590080:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.933618:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.934592:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0890780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280590080:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.934609:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x95:0x0](ffff8ab3f12e1148) 00000080:00200000:2.0:1787827204.934626:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub67, unhashed 1 00000080:00200000:2.0:1787827204.934650:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.934661:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.934671:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.934681:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.934689:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.934694:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.934696:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.934700:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.934704:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.934712:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.934723:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.934728:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.934732:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.934735:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.934737:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.934745:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub68, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.934749:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.934753:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.934757:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub68, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.934760:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub68, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.934785:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.934792:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0892d00 x1874671280590336/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.934806:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0892d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280590336:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.934850:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.935818:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0892d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280590336:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.935835:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x96:0x0](ffff8ab3f12e5b88) 00000080:00200000:2.0:1787827204.935848:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub68, unhashed 1 00000080:00200000:2.0:1787827204.935865:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.935872:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.935879:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.935905:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.935915:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.935920:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.935923:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.935928:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.935932:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.935941:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.935952:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.935957:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.935961:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.935963:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.935965:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.935972:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub69, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.935974:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.935977:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.935979:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub69, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.935982:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub69, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.936003:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.936007:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0890f00 x1874671280590592/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.936018:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0890f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280590592:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.936049:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.937016:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0890f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280590592:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.937055:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x97:0x0](ffff8ab3f13921c8) 00000080:00200000:2.0:1787827204.937068:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub69, unhashed 1 00000080:00200000:2.0:1787827204.937086:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.937104:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.937114:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.937122:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.937127:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.937131:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.937133:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.937137:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.937140:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.937145:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.937152:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.937155:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.937158:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.937159:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.937161:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.937167:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub70, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.937170:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.937172:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.937175:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub70, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.937177:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub70, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.937196:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.937201:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0893840 x1874671280590848/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.937211:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0893840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280590848:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.937240:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.938197:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0893840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280590848:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.938214:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x98:0x0](ffff8ab3f13942c8) 00000080:00200000:2.0:1787827204.938226:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub70, unhashed 1 00000080:00200000:2.0:1787827204.938243:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.938249:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.938256:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.938263:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.938268:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.938271:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.938273:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.938276:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.938278:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.938283:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.938290:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.938292:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.938295:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.938297:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.938299:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.938304:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub71, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.938307:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.938309:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.938312:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub71, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.938314:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub71, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.938331:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.938336:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0893480 x1874671280591104/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.938346:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0893480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280591104:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.938387:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.939402:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0893480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280591104:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.939421:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x99:0x0](ffff8ab3f1396c08) 00000080:00200000:2.0:1787827204.939438:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub71, unhashed 1 00000080:00200000:2.0:1787827204.939460:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.939469:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.939477:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.939488:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.939509:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.939514:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.939518:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.939522:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.939525:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.939533:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.939542:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.939546:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.939551:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.939553:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.939556:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.939566:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub72, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.939568:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.939573:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.939576:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub72, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.939579:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub72, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.939607:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.939614:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d08930c0 x1874671280591360/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.939629:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08930c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280591360:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.939661:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.940598:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d08930c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280591360:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.940615:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x9a:0x0](ffff8ab3f1393a88) 00000080:00200000:2.0:1787827204.940627:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub72, unhashed 1 00000080:00200000:2.0:1787827204.940643:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.940649:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.940656:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.940663:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.940667:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.940670:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.940672:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.940676:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.940678:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.940684:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.940691:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.940693:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.940695:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.940697:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.940699:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.940705:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub73, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.940708:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.940711:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.940713:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub73, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.940715:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub73, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.940732:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.940737:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0891e00 x1874671280591616/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.940747:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0891e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280591616:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.940776:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.941788:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0891e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280591616:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.941805:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x9b:0x0](ffff8ab3f1395348) 00000080:00200000:2.0:1787827204.941829:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub73, unhashed 1 00000080:00200000:2.0:1787827204.941850:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.941860:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.941869:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.941879:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.941906:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.941910:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.941913:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.941916:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.941919:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.941926:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.941933:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.941936:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.941940:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.941943:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.941945:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.941953:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub74, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.941955:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.941958:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.941961:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub74, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.941963:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub74, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.941992:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.941997:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a01e00 x1874671280591872/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.942007:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a01e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280591872:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.942036:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.942992:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a01e00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280591872:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.943007:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x9c:0x0](ffff8ab3f1397448) 00000080:00200000:2.0:1787827204.943019:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub74, unhashed 1 00000080:00200000:2.0:1787827204.943035:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.943041:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.943047:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.943054:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.943059:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.943062:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.943064:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.943067:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.943069:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.943075:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.943081:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.943084:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.943087:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.943089:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.943091:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.943111:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub75, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.943114:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.943117:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.943119:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub75, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.943121:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub75, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.943140:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.943145:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a02940 x1874671280592128/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.943155:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a02940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280592128:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.943183:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.944277:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a02940 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280592128:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.944294:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x9d:0x0](ffff8ab3f13900c8) 00000080:00200000:2.0:1787827204.944306:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub75, unhashed 1 00000080:00200000:2.0:1787827204.944321:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.944327:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.944334:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.944341:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.944346:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.944349:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.944351:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.944355:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.944357:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.944363:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.944369:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.944372:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.944374:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.944376:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.944378:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.944383:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub76, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.944385:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.944388:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.944390:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub76, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.944392:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub76, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.944409:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.944414:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a00b40 x1874671280592384/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.944424:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a00b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280592384:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.944459:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.945473:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a00b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280592384:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.945488:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x9e:0x0](ffff8ab3f1394b08) 00000080:00200000:2.0:1787827204.945500:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub76, unhashed 1 00000080:00200000:2.0:1787827204.945520:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.945530:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.945538:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.945549:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.945557:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.945562:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.945565:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.945569:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.945571:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.945578:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.945584:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.945587:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.945590:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.945591:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.945593:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.945599:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub77, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.945601:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.945604:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.945607:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub77, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.945609:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub77, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.945627:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.945632:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a00000 x1874671280592640/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.945642:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a00000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280592640:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.945671:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.946661:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a00000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280592640:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.946681:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0x9f:0x0](ffff8ab3f13963c8) 00000080:00200000:2.0:1787827204.946693:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub77, unhashed 1 00000080:00200000:2.0:1787827204.946709:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.946715:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.946721:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.946728:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.946733:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.946736:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.946738:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.946742:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.946744:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.946749:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.946756:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.946758:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.946761:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.946763:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.946765:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.946771:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub78, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.946773:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.946776:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.946778:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub78, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.946780:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub78, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.946797:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.946802:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a021c0 x1874671280592896/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.946812:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a021c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280592896:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.946843:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.947822:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a021c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280592896:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.947846:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xa0:0x0](ffff8ab3f1392a08) 00000080:00200000:2.0:1787827204.947865:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub78, unhashed 1 00000080:00200000:2.0:1787827204.947925:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.947937:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.947946:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.947958:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.947966:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.947970:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.947974:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.947980:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.947983:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.947992:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.948002:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.948007:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.948012:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.948015:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.948018:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.948039:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub79, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.948043:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.948047:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.948051:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub79, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.948055:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub79, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.948081:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.948114:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a01680 x1874671280593152/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.948131:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a01680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280593152:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.948192:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.949240:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a01680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280593152:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.949262:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xa1:0x0](ffff8ab3f1391988) 00000080:00200000:2.0:1787827204.949279:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub79, unhashed 1 00000080:00200000:2.0:1787827204.949296:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.949303:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.949311:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.949322:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.949329:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.949334:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.949337:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.949342:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.949346:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.949355:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.949366:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.949371:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.949375:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.949378:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.949381:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.949390:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub80, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.949394:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.949399:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.949402:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub80, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.949406:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub80, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.949430:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.949436:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a03c00 x1874671280593408/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.949455:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a03c00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280593408:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.949483:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.950657:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a03c00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280593408:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.950676:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xa2:0x0](ffff8ab3f1391148) 00000080:00200000:2.0:1787827204.950688:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub80, unhashed 1 00000080:00200000:2.0:1787827204.950714:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.950722:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.950729:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.950736:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.950741:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.950744:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.950746:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.950750:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.950752:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.950758:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.950765:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.950768:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.950771:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.950773:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.950774:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.950780:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub81, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.950783:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.950785:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.950788:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub81, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.950790:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub81, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.950809:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.950814:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a01a40 x1874671280593664/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.950825:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a01a40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280593664:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.950854:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.952003:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a01a40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280593664:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.952021:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xa3:0x0](ffff8ab3f1395b88) 00000080:00200000:2.0:1787827204.952035:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub81, unhashed 1 00000080:00200000:2.0:1787827204.952055:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.952064:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.952074:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.952084:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.952102:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.952108:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.952111:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.952117:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.952122:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.952131:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.952139:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.952142:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.952145:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.952147:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.952149:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.952156:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub82, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.952159:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.952162:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.952164:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub82, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.952167:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub82, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.952188:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.952194:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a012c0 x1874671280593920/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.952206:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a012c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280593920:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.952248:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.953252:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a012c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280593920:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.953278:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xa4:0x0](ffff8ab3f1390908) 00000080:00200000:2.0:1787827204.953294:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub82, unhashed 1 00000080:00200000:2.0:1787827204.953316:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.953324:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.953330:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.953338:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.953342:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.953346:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.953348:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.953351:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.953354:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.953360:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.953367:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.953369:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.953371:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.953373:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.953375:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.953381:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub83, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.953383:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.953386:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.953388:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub83, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.953391:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub83, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.953409:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.953415:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a003c0 x1874671280594176/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.953426:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a003c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280594176:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.953459:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.954447:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a003c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280594176:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.954468:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xa5:0x0](ffff8ab3f1393248) 00000080:00200000:2.0:1787827204.954486:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub83, unhashed 1 00000080:00200000:2.0:1787827204.954518:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.954526:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.954532:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.954540:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.954545:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.954548:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.954550:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.954554:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.954556:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.954562:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.954568:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.954571:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.954574:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.954576:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.954577:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.954584:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub84, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.954586:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.954589:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.954592:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub84, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.954594:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub84, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.954613:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.954617:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a02580 x1874671280594432/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.954628:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a02580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280594432:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.954658:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.955715:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a02580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280594432:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.955755:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xa6:0x0](ffff8ab3f13863c8) 00000080:00200000:2.0:1787827204.955770:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub84, unhashed 1 00000080:00200000:2.0:1787827204.955788:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.955795:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.955802:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.955811:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.955816:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.955819:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.955822:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.955826:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.955828:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.955836:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.955847:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.955852:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.955856:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.955859:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.955862:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.955872:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub85, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.955876:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.955881:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.955904:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub85, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.955908:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub85, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.955930:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.955936:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a00780 x1874671280594688/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.955949:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a00780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280594688:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.955988:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.957184:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a00780 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280594688:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.957211:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xa7:0x0](ffff8ab3f1382a08) 00000080:00200000:2.0:1787827204.957225:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub85, unhashed 1 00000080:00200000:2.0:1787827204.957243:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.957250:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.957258:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.957265:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.957271:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.957274:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.957276:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.957280:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.957282:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.957288:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.957295:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.957298:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.957300:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.957302:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.957303:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.957311:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub86, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.957313:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.957316:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.957318:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub86, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.957321:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub86, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.957340:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.957345:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a02d00 x1874671280594944/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.957356:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a02d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280594944:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.957389:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.958364:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a02d00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280594944:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.958380:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xa8:0x0](ffff8ab3f1381988) 00000080:00200000:2.0:1787827204.958392:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub86, unhashed 1 00000080:00200000:2.0:1787827204.958409:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.958416:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.958422:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.958430:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.958434:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.958437:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.958439:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.958442:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.958446:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.958455:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.958466:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.958470:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.958475:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.958478:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.958481:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.958491:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub87, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.958494:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.958499:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.958501:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub87, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.958503:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub87, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.958533:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.958538:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a00f00 x1874671280595200/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.958549:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a00f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280595200:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.958591:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.959620:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a00f00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280595200:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.959640:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xa9:0x0](ffff8ab3f1381148) 00000080:00200000:2.0:1787827204.959652:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub87, unhashed 1 00000080:00200000:2.0:1787827204.959669:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.959675:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.959682:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.959688:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.959693:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.959696:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.959698:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.959703:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.959706:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.959715:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.959734:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.959740:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.959744:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.959747:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.959751:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.959760:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub88, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.959762:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.959766:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.959768:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub88, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.959770:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub88, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.959791:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.959795:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a03840 x1874671280595456/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.959806:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a03840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280595456:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.959837:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.960962:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a03840 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280595456:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.960983:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xaa:0x0](ffff8ab3f1385b88) 00000080:00200000:2.0:1787827204.960997:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub88, unhashed 1 00000080:00200000:2.0:1787827204.961014:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.961021:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.961028:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.961035:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.961040:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.961043:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.961045:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.961049:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.961051:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.961060:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.961071:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.961076:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.961079:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.961083:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.961086:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.961106:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub89, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.961110:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.961115:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.961118:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub89, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.961120:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub89, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.961141:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.961146:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a03480 x1874671280595712/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.961157:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a03480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280595712:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.961186:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.962183:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a03480 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280595712:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.962201:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xab:0x0](ffff8ab3f1380908) 00000080:00200000:2.0:1787827204.962213:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub89, unhashed 1 00000080:00200000:2.0:1787827204.962232:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.962242:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.962250:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.962260:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.962268:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.962272:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.962275:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.962280:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.962284:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.962293:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.962303:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.962308:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.962313:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.962316:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.962319:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.962328:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub90, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.962332:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.962337:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.962340:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub90, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.962344:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub90, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.962366:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.962371:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0a030c0 x1874671280595968/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.962382:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a030c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280595968:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.962412:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.963442:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a030c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280595968:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.963463:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xac:0x0](ffff8ab3f1383248) 00000080:00200000:2.0:1787827204.963481:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub90, unhashed 1 00000080:00200000:2.0:1787827204.963503:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.963523:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.963532:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.963543:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.963551:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.963555:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.963558:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.963562:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.963564:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.963570:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.963576:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.963579:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.963582:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.963584:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.963585:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.963591:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub91, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.963593:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.963596:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.963598:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub91, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.963601:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub91, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.963635:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.963640:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0bd4b40 x1874671280596224/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.963651:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0bd4b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280596224:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.963680:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.964718:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0bd4b40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280596224:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.964750:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xad:0x0](ffff8ab3f13821c8) 00000080:00200000:2.0:1787827204.964766:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub91, unhashed 1 00000080:00200000:2.0:1787827204.964783:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.964790:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.964796:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.964803:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.964808:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.964811:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.964813:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.964816:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.964819:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.964825:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.964832:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.964836:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.964839:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.964842:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.964845:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.964853:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub92, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.964857:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.964861:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.964865:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub92, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.964869:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub92, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.964910:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.964915:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0bd4000 x1874671280596480/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.964926:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0bd4000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280596480:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.964961:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.966114:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0bd4000 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280596480:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.966130:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xae:0x0](ffff8ab3f13842c8) 00000080:00200000:2.0:1787827204.966142:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub92, unhashed 1 00000080:00200000:2.0:1787827204.966159:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.966166:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.966172:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.966179:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.966184:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.966187:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.966189:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.966192:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.966194:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.966200:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.966206:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.966209:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.966212:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.966214:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.966216:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.966223:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub93, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.966225:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.966227:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.966230:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub93, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.966232:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub93, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.966249:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.966254:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0bd61c0 x1874671280596736/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.966264:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0bd61c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280596736:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.966309:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.967206:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0bd61c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280596736:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.967222:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xaf:0x0](ffff8ab3f1386c08) 00000080:00200000:2.0:1787827204.967246:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub93, unhashed 1 00000080:00200000:2.0:1787827204.967262:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.967268:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.967275:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.967282:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.967286:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.967289:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.967291:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.967295:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.967297:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.967303:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.967309:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.967312:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.967314:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.967316:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.967318:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.967324:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub94, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.967326:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.967329:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.967331:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub94, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.967333:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub94, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.967350:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.967355:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0bd5680 x1874671280596992/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.967365:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0bd5680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280596992:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.967394:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.968360:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0bd5680 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280596992:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.968377:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xb0:0x0](ffff8ab3f1383a88) 00000080:00200000:2.0:1787827204.968388:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub94, unhashed 1 00000080:00200000:2.0:1787827204.968404:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.968411:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.968418:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.968425:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.968429:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.968432:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.968433:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.968436:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.968439:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.968446:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.968457:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.968461:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.968465:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.968469:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.968472:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.968482:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub95, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.968485:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.968490:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.968494:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub95, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.968497:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub95, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.968523:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.968531:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0bd7c00 x1874671280597248/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.968545:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0bd7c00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280597248:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.968595:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.969600:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0bd7c00 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280597248:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.969616:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xb1:0x0](ffff8ab3f1385348) 00000080:00200000:2.0:1787827204.969628:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub95, unhashed 1 00000080:00200000:2.0:1787827204.969644:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.969651:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.969657:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.969664:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.969669:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.969672:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.969673:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.969677:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.969679:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.969684:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.969691:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.969693:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.969696:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.969698:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.969700:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.969708:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub96, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.969712:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.969716:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.969720:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub96, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.969733:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub96, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.969759:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.969764:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0bd5a40 x1874671280597504/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.969775:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0bd5a40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280597504:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.969806:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.970858:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0bd5a40 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280597504:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.970878:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xb2:0x0](ffff8ab3f1387448) 00000080:00200000:2.0:1787827204.970941:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub96, unhashed 1 00000080:00200000:2.0:1787827204.970962:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.970970:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.970978:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.970986:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.970992:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.970996:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.970998:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.971002:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.971005:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.971011:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.971018:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.971020:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.971023:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.971025:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.971027:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.971033:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub97, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.971036:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.971039:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.971041:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub97, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.971044:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub97, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.971064:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.971069:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0bd52c0 x1874671280597760/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.971081:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0bd52c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280597760:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.971129:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.972137:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0bd52c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280597760:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.972154:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xb3:0x0](ffff8ab3f13800c8) 00000080:00200000:2.0:1787827204.972166:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub97, unhashed 1 00000080:00200000:2.0:1787827204.972184:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.972190:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.972197:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.972205:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.972210:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.972213:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.972215:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.972218:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.972220:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.972234:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.972241:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.972245:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.972248:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.972252:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.972255:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.972263:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub98, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.972267:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.972271:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.972275:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub98, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.972279:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub98, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.972305:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.972310:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0bd43c0 x1874671280598016/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.972321:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0bd43c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280598016:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.972350:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.973366:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0bd43c0 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280598016:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.973385:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xb4:0x0](ffff8ab3f1384b08) 00000080:00200000:2.0:1787827204.973399:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub98, unhashed 1 00000080:00200000:2.0:1787827204.973418:0:48446:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827204.973425:0:48446:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827204.973432:0:48446:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.973441:0:48446:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.973446:0:48446:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827204.973450:0:48446:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827204.973452:0:48446:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827204.973455:0:48446:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e87e93bd0 00010000:00010000:2.0:1787827204.973458:0:48446:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827204.973464:0:48446:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827204.973471:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.973474:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d6.sanity-scrub, flags=24 00000080:00200000:2.0:1787827204.973477:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200001b72:0x1:0x0](ffff8ab3f139b248), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.973479:0:48446:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1787827204.973481:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 1 00000080:00200000:2.0:1787827204.973487:0:48446:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f6.sanity-scrub99, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=1536 00000080:00200000:2.0:1787827204.973489:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.973492:0:48446:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), inode mode 41ed mask 3 00000080:00200000:2.0:1787827204.973494:0:48446:0:(namei.c:2129:ll_create_nd()) VFS Op:name=f6.sanity-scrub99, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148), flags=33060, excl=1 00000080:00200000:2.0:1787827204.973497:0:48446:0:(namei.c:2084:ll_mknod()) VFS Op:name=f6.sanity-scrub99, dir=[0x240001b71:0x1:0x0](ffff8ab3d1f49148) mode 100444 dev 0 00000020:00100000:2.0:1787827204.973523:0:48446:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff8ab3c584d000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827204.973531:0:48446:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8ab3d0bd6580 x1874671280598272/t0(0) o36->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827204.973558:0:48446:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0bd6580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280598272:192.168.203.124@tcp:36:createmany.0 00000100:00100000:2.0:1787827204.973592:0:48446:0:(client.c:2715:ptlrpc_set_wait()) set ffff8ab3c79130c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827204.974610:0:48446:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0bd6580 pname:cluuid:pid:xid:nid:opc:job createmany:27aedc06-3022-4c11-aafb-6bd27e69b836:48446:1874671280598272:192.168.203.124@tcp:36:createmany.0 00000080:00200000:2.0:1787827204.974647:0:48446:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240001b71:0xb5:0x0](ffff8ab3f13880c8) 00000080:00200000:2.0:1787827204.974662:0:48446:0:(namei.c:2137:ll_create_nd()) VFS Op:name=f6.sanity-scrub99, unhashed 1 00000080:00200008:2.0:1787827205.085408:0:48484:0:(file.c:6313:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8ab3f137a1c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827205.085422:0:48484:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:2.0:1787827205.085449:0:48484:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827205.085469:0:48484:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827205.085484:0:48484:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827205.085491:0:48484:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827205.085499:0:48484:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:2.0:1787827205.085503:0:48484:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:2.0:1787827205.085511:0:48484:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e817c3cb8 00010000:00010000:2.0:1787827205.085514:0:48484:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827205.085522:0:48484:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200008:0.0:1787827205.292344:0:48485:0:(file.c:6313:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8ab3f137a1c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827205.292355:0:48485:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827205.292373:0:48485:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827205.292389:0:48485:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827205.292401:0:48485:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827205.292407:0:48485:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827205.292413:0:48485:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827205.292416:0:48485:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827205.292422:0:48485:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e817c3cb8 00010000:00010000:0.0:1787827205.292425:0:48485:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827205.292432:0:48485:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200008:0.0:1787827205.292525:0:48485:0:(file.c:6313:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8ab3f137a1c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827205.292529:0:48485:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8ab3f137a1c8),name=/ 00000002:00010000:0.0:1787827205.292536:0:48485:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827205.292541:0:48485:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827205.292548:0:48485:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1787827205.292552:0:48485:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827205.292555:0:48485:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) 00000080:00010000:0.0:1787827205.292557:0:48485:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8ab3f137a1c8) for lock 0x268e1a602cc1004 00000080:00010000:0.0:1787827205.292560:0:48485:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x268e1a602cc1004 from it ffffb86e817c3cb8 00010000:00010000:0.0:1787827205.292562:0:48485:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1787827205.292567:0:48485:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d800/0x268e1a602cc1004 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x772721ad1f0d6a0e expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1787827205.300279:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.300308:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.300344:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000004cf52578 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000085e596c3/0x268e1a602cc105f lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6b42 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000008:00010000:3.0:1787827205.300364:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000004cf52578 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000085e596c3/0x268e1a602cc105f lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6b42 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:3.0:1787827205.300425:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000085e596c3/0x268e1a602cc105f lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6b42 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00100000:3.0:1787827205.300530:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3e0ba4b40 - object 0x2c0000401:98 offset 0<>903 00000080:00200000:3.0:1787827205.300576:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.300587:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000100:00100000:0.0:1787827205.300593:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0ba4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280598400:192.168.203.124@tcp:4:cp.0 00020000:00200000:3.0:1787827205.300596:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787827205.300608:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.300615:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.300618:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.300625:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000c4744fd2 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc10ac lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->393215) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6c14 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.300634:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000c4744fd2 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc10ac lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->393215) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6c14 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.300782:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc10ac lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->393215) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6c14 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00100000:3.0:1787827205.301329:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3e0ba6d00 - object 0x2c0000401:99 offset 0<>391609 00000080:00200000:3.0:1787827205.301368:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1787827205.301378:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0ba6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280598528:192.168.203.124@tcp:4:cp.0 00000080:00200000:3.0:1787827205.301380:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.301389:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787827205.301400:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.301409:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.301413:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.301421:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000005e800391 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000003afe7a08/0x268e1a602cc10f2 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6cca expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.301432:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000005e800391 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000003afe7a08/0x268e1a602cc10f2 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6cca expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.301465:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000003afe7a08/0x268e1a602cc10f2 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6cca expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.301520:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3e0ba70c0 - object 0x2c0000401:100 offset 0<>9974 00000080:00200000:3.0:1787827205.301554:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.301564:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.301570:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787827205.301635:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.301642:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.301646:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.301653:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000f6599e65 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000007c8a0dd5/0x268e1a602cc1138 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6d80 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.301661:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000f6599e65 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000007c8a0dd5/0x268e1a602cc1138 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6d80 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.301700:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000007c8a0dd5/0x268e1a602cc1138 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6d80 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.301755:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3e0ba7840 - object 0x2c0000401:101 offset 0<>39343 00000080:00200000:3.0:1787827205.301780:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.301787:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.301791:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787827205.301798:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.301810:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0xb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.301816:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.301878:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000007fb8f05a ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c170f3e3/0x268e1a602cc117e lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6e36 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.301922:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000007fb8f05a ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c170f3e3/0x268e1a602cc117e lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6e36 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.301950:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c170f3e3/0x268e1a602cc117e lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6e36 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.301985:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3e0ba5680 - object 0x2c0000401:102 offset 0<>674 00000100:00100000:0.0:1787827205.302026:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0ba5680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280598912:192.168.203.124@tcp:4:cp.0 00000080:00200000:3.0:1787827205.302064:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0xb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.302078:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.302084:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787827205.302110:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.302125:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0xd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.302129:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.302140:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000004bb409dd ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d35f58b8/0x268e1a602cc11c4 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6eec expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.302153:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000004bb409dd ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d35f58b8/0x268e1a602cc11c4 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6eec expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.302189:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d35f58b8/0x268e1a602cc11c4 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6eec expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.302226:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3e0ba6580 - object 0x2c0000401:103 offset 0<>2659 00000100:00100000:0.0:1787827205.302282:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0ba6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280599040:192.168.203.124@tcp:4:cp.0 00000080:00200000:3.0:1787827205.302604:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0xd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.302667:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0xe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.302674:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787827205.302687:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0xe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.302698:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0xf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.302703:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.302714:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000000fb4eddb ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000116384d8/0x268e1a602cc120a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6fa2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.302726:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000004cf52578 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000085e596c3/0x268e1a602cc105f lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6b42 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00010000:0.0:1787827205.302773:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000004cf52578 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000085e596c3/0x268e1a602cc105f lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6b42 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000100:00100000:0.0:1787827205.302806:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280598400:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827205.303047:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000000fb4eddb ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000116384d8/0x268e1a602cc120a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6fa2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.303208:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000116384d8/0x268e1a602cc120a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6fa2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.303264:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3e0ba4f00 - object 0x2c0000401:104 offset 0<>28245 00000080:00200000:3.0:1787827205.303409:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0xf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.303431:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x10:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.303439:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1787827205.303755:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000004bb409dd ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d35f58b8/0x268e1a602cc11c4 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6eec expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.303773:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000004bb409dd ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d35f58b8/0x268e1a602cc11c4 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6eec expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827205.303790:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280599040:192.168.203.124@tcp:4:cp.0 00000080:00200000:3.0:1787827205.303836:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x10:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.303850:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x11:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.303862:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.303871:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000097c6dcd2 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005dcc3b5d/0x268e1a602cc1250 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7058 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.303880:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000097c6dcd2 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005dcc3b5d/0x268e1a602cc1250 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7058 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.303927:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005dcc3b5d/0x268e1a602cc1250 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7058 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.303956:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3e0ba7480 - object 0x2c0000401:105 offset 0<>1754 00000080:00200000:3.0:1787827205.303977:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x11:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1787827205.303990:0:10955:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000080:00200000:3.0:1787827205.303992:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x12:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.303998:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1787827205.304005:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0ba7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280599296:192.168.203.124@tcp:4:cp.0 00000080:00200000:3.0:1787827205.304061:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x12:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.304069:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x13:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.304074:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.304082:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000039dfb438 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000369f72cf/0x268e1a602cc1296 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d710e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.304104:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000039dfb438 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000369f72cf/0x268e1a602cc1296 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d710e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.304124:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000369f72cf/0x268e1a602cc1296 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d710e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.304159:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3e0ba6940 - object 0x2c0000401:106 offset 0<>1613 00000080:00200000:3.0:1787827205.304182:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x13:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1787827205.304191:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0ba6940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280599424:192.168.203.124@tcp:4:cp.0 00000080:00200000:3.0:1787827205.304191:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x14:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.304202:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1787827205.304211:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x14:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.304223:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x15:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.304226:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.304233:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000004eb90899 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b209a86c/0x268e1a602cc12dc lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d71c4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.304242:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000004eb90899 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b209a86c/0x268e1a602cc12dc lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d71c4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.304259:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000b209a86c/0x268e1a602cc12dc lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d71c4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.304285:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3e0ba43c0 - object 0x2c0000401:107 offset 0<>838 00000080:00200000:3.0:1787827205.304384:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x15:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.304397:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x16:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000100:00100000:0.0:1787827205.304412:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3e0ba43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280599552:192.168.203.124@tcp:4:cp.0 00020000:00200000:3.0:1787827205.304611:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.304632:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000094ceebfe ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b6631250/0x268e1a602cc12ff lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d721f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.304640:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000094ceebfe ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b6631250/0x268e1a602cc12ff lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d721f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.304659:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000b6631250/0x268e1a602cc12ff lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d721f expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.304863:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3e0ba5e00 - object 0x280000401:107 offset 0<>2858 00000080:00200000:3.0:1787827205.304912:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x16:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.304966:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x17:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.304971:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.304978:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000034c916b1 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e0bdd12f/0x268e1a602cc1322 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->20479) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d727a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.304987:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000034c916b1 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e0bdd12f/0x268e1a602cc1322 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->20479) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d727a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.305017:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e0bdd12f/0x268e1a602cc1322 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->20479) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d727a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.305054:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a40000 - object 0x2c0000401:108 offset 0<>18269 00000080:00200000:3.0:1787827205.305077:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x17:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.305086:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x18:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.305105:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1787827205.305295:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000007fb8f05a ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c170f3e3/0x268e1a602cc117e lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6e36 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.305318:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000007fb8f05a ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c170f3e3/0x268e1a602cc117e lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6e36 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1787827205.305330:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000002fe1cb0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000da540131/0x268e1a602cc1345 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d72d5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.305341:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000002fe1cb0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000da540131/0x268e1a602cc1345 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d72d5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787827205.305349:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba5680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280598912:192.168.203.124@tcp:4:cp.0 00010000:00010000:3.0:1787827205.305480:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000da540131/0x268e1a602cc1345 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d72d5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.305565:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a403c0 - object 0x280000401:108 offset 0<>4218 00000080:00200000:3.0:1787827205.305587:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x18:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1787827205.305599:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x19:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.305604:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.305613:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000ee8f0f8f ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050992227/0x268e1a602cc1368 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7330 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787827205.305615:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280599936:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827205.305638:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000ee8f0f8f ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050992227/0x268e1a602cc1368 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7330 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.305658:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050992227/0x268e1a602cc1368 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7330 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.305687:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a43c00 - object 0x2c0000401:109 offset 0<>3006 00000080:00200000:3.0:1787827205.305720:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x19:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1787827205.305737:0:10957:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:0.0:1787827205.305753:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280600064:192.168.203.124@tcp:4:cp.0 00000080:00200000:3.0:1787827205.305955:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x1a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.305966:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.305982:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000006c0f17f1 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc138b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d738b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.305996:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000006c0f17f1 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc138b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d738b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.306205:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc138b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d738b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.306287:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a412c0 - object 0x280000401:109 offset 0<>2192 00000080:00200000:3.0:1787827205.306321:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x1a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:0.0:1787827205.306413:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000c4744fd2 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc10ac lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->393215) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6c14 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1787827205.306462:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x1b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.306471:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.306495:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000007431681f ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fb353673/0x268e1a602cc13ae lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d73e6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.306523:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000c4744fd2 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000900cd124/0x268e1a602cc10ac lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->393215) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d6c14 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000100:00100000:0.0:1787827205.306551:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280598528:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827205.306555:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000007431681f ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fb353673/0x268e1a602cc13ae lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d73e6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.306589:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fb353673/0x268e1a602cc13ae lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d73e6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.306626:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a421c0 - object 0x2c0000401:110 offset 0<>938 00000080:00200000:3.0:1787827205.306658:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x1b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1787827205.306675:0:10955:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000080:00200000:3.0:1787827205.306679:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x1c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1787827205.306688:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1787827205.306695:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280600320:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827205.306700:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000013fc7cd2 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002dea2600/0x268e1a602cc13d1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->57343) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7441 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.306711:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000013fc7cd2 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002dea2600/0x268e1a602cc13d1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->57343) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7441 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.306759:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002dea2600/0x268e1a602cc13d1 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->57343) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7441 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.307053:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0bd4f00 - object 0x280000401:110 offset 0<>55540 00000080:00200000:2.0:1787827205.307118:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x1c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.307136:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.307143:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1787827205.307151:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0bd4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280600448:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.307156:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000b805e6dd ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f2dfd3db/0x268e1a602cc13f4 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d749c expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.307179:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000b805e6dd ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f2dfd3db/0x268e1a602cc13f4 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d749c expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.307229:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f2dfd3db/0x268e1a602cc13f4 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d749c expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.307280:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0bd7840 - object 0x2c0000401:111 offset 0<>6813 00000080:00200000:2.0:1787827205.307312:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.307327:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x1e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.307333:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1787827205.307344:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000df3b1593 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000025023211/0x268e1a602cc1417 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d74f7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.307358:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000df3b1593 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000025023211/0x268e1a602cc1417 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d74f7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.307379:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000025023211/0x268e1a602cc1417 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d74f7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.307410:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0bd7480 - object 0x280000401:111 offset 0<>6213 00000080:00200000:2.0:1787827205.307438:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x1e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.307451:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x1f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.307455:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.307468:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x1f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1787827205.307468:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0bd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280600704:192.168.203.124@tcp:4:cp.0 00000080:00200000:2.0:1787827205.307479:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x20:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.307490:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1787827205.307497:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000dbe12435 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e429a3c3/0x268e1a602cc145d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d75ad expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.307504:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000dbe12435 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e429a3c3/0x268e1a602cc145d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d75ad expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.307522:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e429a3c3/0x268e1a602cc145d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d75ad expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.307543:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0bd70c0 - object 0x280000401:112 offset 0<>234 00000080:00200000:2.0:1787827205.307577:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x20:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.307584:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x21:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.307591:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.307600:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x21:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.307610:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x22:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.307615:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1787827205.307625:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000003613c446 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f0ec90a/0x268e1a602cc14a3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7663 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.307639:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000003613c446 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f0ec90a/0x268e1a602cc14a3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7663 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.307667:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f0ec90a/0x268e1a602cc14a3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7663 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.307717:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0bd5e00 - object 0x280000401:113 offset 0<>3079 00000080:00200000:2.0:1787827205.307744:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x22:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.307762:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x23:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.307771:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.307779:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x23:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.307789:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x24:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.307793:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1787827205.307799:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000009ced413a ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e4c79328/0x268e1a602cc14e9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7719 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.307807:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000009ced413a ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e4c79328/0x268e1a602cc14e9 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7719 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.307836:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e4c79328/0x268e1a602cc14e9 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7719 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.307881:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0bd6940 - object 0x280000401:114 offset 0<>7187 00000080:00200000:2.0:1787827205.307940:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x24:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.307952:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x25:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.307961:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.307971:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x25:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.307977:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x26:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.307981:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1787827205.307989:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000004a4504da ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000010f3849a/0x268e1a602cc152f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d77cf expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.307997:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000004a4504da ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000010f3849a/0x268e1a602cc152f lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d77cf expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.308019:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000010f3849a/0x268e1a602cc152f lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d77cf expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.308052:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0bd6d00 - object 0x280000401:115 offset 0<>1790 00000080:00200000:2.0:1787827205.308071:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x26:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308106:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x27:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308112:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308126:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x27:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308137:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x28:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308141:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1787827205.308150:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000007768a513 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c7c240d6/0x268e1a602cc1575 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7885 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.308164:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000007768a513 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c7c240d6/0x268e1a602cc1575 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7885 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.308192:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c7c240d6/0x268e1a602cc1575 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7885 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.308222:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0bd4780 - object 0x280000401:116 offset 0<>6430 00000080:00200000:2.0:1787827205.308241:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x28:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308249:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x29:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308253:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308262:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x29:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308273:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x2a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308280:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308288:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x2a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:3.0:1787827205.308296:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000006c0f17f1 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc138b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d738b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827205.308297:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x2b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308307:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308322:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x2b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308331:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x2c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308337:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308347:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x2c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308355:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x2d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308360:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308367:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x2d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308383:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x2e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308390:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1787827205.308392:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000006c0f17f1 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005d9d72da/0x268e1a602cc138b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d738b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827205.308403:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x2e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308414:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x2f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308420:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1787827205.308424:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280600192:192.168.203.124@tcp:4:cp.0 00000080:00200000:2.0:1787827205.308431:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x2f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308440:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x30:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308444:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1787827205.308452:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000215cb624 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082a8b406/0x268e1a602cc15bb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->114687) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d793b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.308464:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000215cb624 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082a8b406/0x268e1a602cc15bb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->114687) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d793b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.308534:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000082a8b406/0x268e1a602cc15bb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->114687) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d793b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1787827205.308557:0:14753:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000f02bcfcf ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000066b0e2f9/0x268e1a602cc1647 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->163839) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7aa7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.308571:0:14753:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000f02bcfcf ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000066b0e2f9/0x268e1a602cc1647 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->163839) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7aa7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:2.0:1787827205.308645:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a3b0c0 - object 0x280000401:117 offset 0<>111139 00010000:00010000:3.0:1787827205.308655:0:14753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000066b0e2f9/0x268e1a602cc1647 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->163839) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7aa7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827205.308673:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x30:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308685:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x31:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308692:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308706:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x31:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308717:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x32:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308722:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308730:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x32:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308736:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x33:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308739:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308745:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x33:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308752:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x34:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308757:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308763:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x34:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308769:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x35:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308771:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308778:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x35:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308783:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x36:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000008:00100000:3.0:1787827205.308784:0:14753:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a430c0 - object 0x280000401:119 offset 0<>159776 00020000:00200000:2.0:1787827205.308787:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308795:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x36:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308800:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x37:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308802:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308810:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x37:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308824:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x38:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308832:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1787827205.308836:0:10957:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000080:00200000:2.0:1787827205.308849:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x38:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1787827205.308855:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280601600:192.168.203.124@tcp:4:cp.0 00000080:00200000:2.0:1787827205.308862:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x39:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308867:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308875:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x39:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308880:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x3a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308883:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308909:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x3a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308915:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x3b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308918:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308925:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x3b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308930:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x3c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308933:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308939:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x3c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308953:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x3d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308957:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308965:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x3d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.308976:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x3e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.308980:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.308988:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x3e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309002:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x3f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309006:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309017:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x3f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309024:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x40:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309026:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309037:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x40:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309047:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x41:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309050:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309058:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x41:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309065:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x42:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309069:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309076:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x42:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309088:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x43:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309106:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309118:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x43:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309133:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x44:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309137:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309149:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x44:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309159:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x45:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309162:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309168:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x45:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309177:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x46:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309180:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309191:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x46:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309201:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x47:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309204:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309215:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x47:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309224:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x48:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309228:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309240:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x48:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309259:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x49:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309267:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309281:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x49:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309297:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x4a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309302:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309309:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x4a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309315:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x4b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309318:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309327:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x4b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309335:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x4c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309337:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309343:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x4c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309350:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x4d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309353:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309360:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x4d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309369:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x4e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309372:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309380:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x4e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309392:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x4f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309396:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309404:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x4f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309411:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x51:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309415:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309421:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x51:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309430:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x52:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309433:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309443:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x52:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309456:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309459:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309468:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309476:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309479:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309488:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309496:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309499:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309505:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309514:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309517:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309526:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309536:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309539:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309547:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309559:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309562:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309570:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309585:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309589:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309596:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309612:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309616:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309629:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309646:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309653:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309665:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309684:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0xb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309691:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309708:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0xb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309726:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309732:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309744:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309758:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0xd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309763:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309775:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0xd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309784:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0xe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309789:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309799:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0xe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309814:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0xf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309819:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309832:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0xf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309844:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x10:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309849:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309860:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x10:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309873:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x11:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309878:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309932:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x11:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309952:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x12:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309957:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309969:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x12:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.309981:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x13:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.309987:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.309999:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x13:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.310010:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x14:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.310014:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.310024:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x14:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.310043:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x15:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.310047:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.310055:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x15:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.310069:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x16:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.310073:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.310083:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x16:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.310114:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x17:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.310123:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.310135:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x17:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.310149:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x18:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.310153:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.310162:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x18:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.310173:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x19:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.310179:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.310191:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x19:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.310208:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x1a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.310214:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.310230:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x1a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.310255:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x1b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.310263:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.310276:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x1b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.310295:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x1c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.310300:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1787827205.310311:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000004f7cf2d0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc14c6 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d76be expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.310323:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000004f7cf2d0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc14c6 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d76be expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.310356:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc14c6 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d76be expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.310404:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a39e00 - object 0x2c0000401:114 offset 0<>4150 00000008:00010000:3.0:1787827205.310410:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000df3b1593 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000025023211/0x268e1a602cc1417 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d74f7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.310431:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000c76da257 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000baa403a3/0x268e1a602cc19b2 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d838a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.310448:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000c76da257 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000baa403a3/0x268e1a602cc19b2 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d838a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.310450:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000df3b1593 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000025023211/0x268e1a602cc1417 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d74f7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827205.310488:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000baa403a3/0x268e1a602cc19b2 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d838a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1787827205.310493:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0bd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280600704:192.168.203.124@tcp:4:cp.0 00000008:00100000:2.0:1787827205.310540:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a3a940 - object 0x2c0000401:132 offset 0<>26716 00000008:00010000:3.0:1787827205.310555:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000ee8f0f8f ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050992227/0x268e1a602cc1368 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7330 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827205.310574:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x1c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.310583:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.310587:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.310600:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.310617:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x1e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.310623:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.310635:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x1e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:0.0:1787827205.310650:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000007431681f ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fb353673/0x268e1a602cc13ae lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d73e6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827205.310650:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x1f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.310660:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.310678:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x1f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:0.0:1787827205.310690:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000007431681f ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000fb353673/0x268e1a602cc13ae lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d73e6 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827205.310694:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x20:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000008:00010000:3.0:1787827205.310704:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000ee8f0f8f ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000050992227/0x268e1a602cc1368 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7330 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00020000:00200000:2.0:1787827205.310710:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.310731:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x20:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.310757:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x21:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.310765:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1787827205.310765:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280600064:192.168.203.124@tcp:4:cp.0 00000100:00100000:0.0:1787827205.310771:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280600320:192.168.203.124@tcp:4:cp.0 00000080:00200000:2.0:1787827205.310779:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x21:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1787827205.310792:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a39e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280601728:192.168.203.124@tcp:4:cp.0 00000080:00200000:2.0:1787827205.310800:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x22:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.310809:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1787827205.310824:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000080286aa3 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000004941a79b/0x268e1a602cc1a84 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d85ac expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.310835:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000002fe1cb0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000da540131/0x268e1a602cc1345 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d72d5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.310854:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000080286aa3 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000004941a79b/0x268e1a602cc1a84 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d85ac expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.310880:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000002fe1cb0 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000da540131/0x268e1a602cc1345 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d72d5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827205.310921:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000004941a79b/0x268e1a602cc1a84 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d85ac expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.310969:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a38b40 - object 0x2c0000401:135 offset 0<>1765 00000008:00010000:2.0:1787827205.311002:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000048082e40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d935c527/0x268e1a602cc1480 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7608 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.311030:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000048082e40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d935c527/0x268e1a602cc1480 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7608 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787827205.311039:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280599936:192.168.203.124@tcp:4:cp.0 00010000:00010000:2.0:1787827205.311082:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d935c527/0x268e1a602cc1480 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7608 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.311104:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000068cfa346 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c169fc17/0x268e1a602cc1aca lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8662 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.311126:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000068cfa346 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c169fc17/0x268e1a602cc1aca lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8662 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:2.0:1787827205.311137:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a38000 - object 0x2c0000401:113 offset 0<>4117 00000008:00010000:2.0:1787827205.311160:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000aca244fc ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000025e3be85/0x268e1a602cc1782 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7dda expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.311170:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000aca244fc ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000025e3be85/0x268e1a602cc1782 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7dda expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.311178:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c169fc17/0x268e1a602cc1aca lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8662 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.311189:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280602240:192.168.203.124@tcp:4:cp.0 00000008:00100000:0.0:1787827205.311281:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aaf480 - object 0x2c0000401:136 offset 0<>4512 00010000:00010000:2.0:1787827205.311288:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000025e3be85/0x268e1a602cc1782 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7dda expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.311316:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000009a352ff6 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000be9bb69c/0x268e1a602cc1b64 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8dd2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:2.0:1787827205.311336:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a3a1c0 - object 0x2c0000401:124 offset 0<>6548 00000008:00010000:0.0:1787827205.311340:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000009a352ff6 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000be9bb69c/0x268e1a602cc1b64 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8dd2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827205.311359:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x22:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.311371:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x23:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.311377:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00010000:00010000:0.0:1787827205.311386:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000be9bb69c/0x268e1a602cc1b64 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8dd2 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.311387:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000049806261 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005db90aac/0x268e1a602cc16d3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x79:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7c13 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.311427:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000049806261 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005db90aac/0x268e1a602cc16d3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x79:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7c13 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.311454:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005db90aac/0x268e1a602cc16d3 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x79:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7c13 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.311476:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aaed00 - object 0x280000400:130 offset 0<>5634 00000008:00100000:2.0:1787827205.311485:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a39680 - object 0x280000401:121 offset 0<>1659 00000080:00200000:2.0:1787827205.311510:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x23:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1787827205.311537:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a39680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280602752:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.311577:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000039dfb438 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000369f72cf/0x268e1a602cc1296 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d710e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827205.311603:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x24:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000008:00010000:0.0:1787827205.311608:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000039dfb438 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000369f72cf/0x268e1a602cc1296 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d710e expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00020000:00200000:2.0:1787827205.311612:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.311630:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x24:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.311642:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x25:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000100:00100000:0.0:1787827205.311644:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba6940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280599424:192.168.203.124@tcp:4:cp.0 00020000:00200000:2.0:1787827205.311650:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.311665:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x25:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.311675:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x26:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.311681:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1787827205.311682:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000097c6dcd2 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005dcc3b5d/0x268e1a602cc1250 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7058 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827205.311692:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x26:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:0.0:1787827205.311704:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000097c6dcd2 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000005dcc3b5d/0x268e1a602cc1250 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7058 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827205.311706:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x27:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.311721:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.311736:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x27:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1787827205.311742:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3e0ba7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280599296:192.168.203.124@tcp:4:cp.0 00000080:00200000:2.0:1787827205.311749:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x28:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.311756:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1787827205.311767:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000c448afe1 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000001c18fd01/0x268e1a602cc1b87 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->393215) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8e2d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.311772:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000d57973d4 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000001055721d/0x268e1a602cc150c lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7774 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.311778:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000c448afe1 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000001c18fd01/0x268e1a602cc1b87 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->393215) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8e2d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.311813:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000d57973d4 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000001055721d/0x268e1a602cc150c lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7774 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.311931:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000001055721d/0x268e1a602cc150c lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7774 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.312008:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aac780 - object 0x2c0000401:115 offset 0<>8441 00010000:00010000:2.0:1787827205.312059:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000001c18fd01/0x268e1a602cc1b87 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->393215) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8e2d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1787827205.312062:0:10954:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000003613c446 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f0ec90a/0x268e1a602cc14a3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7663 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827205.312076:0:10955:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1787827205.312112:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aac780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280602880:192.168.203.124@tcp:4:cp.0 00000008:00100000:2.0:1787827205.312285:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a3bc00 - object 0x2c0000400:131 offset 0<>391609 00000080:00200000:2.0:1787827205.312319:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x28:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1787827205.312336:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a3bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280603008:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827205.312367:0:10954:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000003613c446 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000001f0ec90a/0x268e1a602cc14a3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7663 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827205.312395:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x29:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.312401:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.312411:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x29:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.312420:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.312423:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.312429:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.312438:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.312441:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.312448:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.312456:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.312459:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.312467:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.312474:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.312477:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.312486:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.312494:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.312497:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.312503:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.312510:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x2f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.312517:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.312529:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x2f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.312536:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x30:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.312539:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.312546:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x30:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.312557:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x31:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.312562:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1787827205.312568:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000eda31fef ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a04a890f/0x268e1a602cc1cc2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9160 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.312585:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000eda31fef ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a04a890f/0x268e1a602cc1cc2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9160 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.312612:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a04a890f/0x268e1a602cc1cc2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9160 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.312658:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a39a40 - object 0x280000400:135 offset 0<>4207 00000008:00010000:2.0:1787827205.312694:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000053d06997 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc1949 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8279 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.312707:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000053d06997 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc1949 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8279 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787827205.312723:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0bd5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280600960:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827205.312772:0:10954:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000007768a513 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c7c240d6/0x268e1a602cc1575 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7885 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.312828:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000013fc7cd2 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002dea2600/0x268e1a602cc13d1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->57343) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7441 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1787827205.312876:0:10954:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000007768a513 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c7c240d6/0x268e1a602cc1575 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7885 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.312883:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000013fc7cd2 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002dea2600/0x268e1a602cc13d1 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->57343) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7441 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1787827205.312924:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0bd4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280601344:192.168.203.124@tcp:4:cp.0 00000100:00100000:2.0:1787827205.312954:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0bd4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280600448:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827205.312961:0:14753:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000c57ffcce ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000098968f12/0x268e1a602cc1d08 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9216 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.312980:0:14753:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000c57ffcce ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000098968f12/0x268e1a602cc1d08 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9216 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.313125:0:14753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000098968f12/0x268e1a602cc1d08 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9216 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827205.313133:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000124c9e21/0x268e1a602cc1949 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8279 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.313173:0:14753:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a43840 - object 0x280000400:136 offset 0<>704 00000008:00010000:3.0:1787827205.313213:0:14753:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000be27f674 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dbb6ce8b/0x268e1a602cc1719 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7cc9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.313229:0:14753:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000be27f674 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dbb6ce8b/0x268e1a602cc1719 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7cc9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.313262:0:14753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dbb6ce8b/0x268e1a602cc1719 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7cc9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.313274:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a392c0 - object 0x280000401:130 offset 0<>255050 00000008:00010000:2.0:1787827205.313306:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000851036a7 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000799c7467/0x268e1a602cc1d4e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d92cc expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.313317:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000851036a7 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000799c7467/0x268e1a602cc1d4e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d92cc expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1787827205.313339:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280603392:192.168.203.124@tcp:4:cp.0 00000008:00100000:3.0:1787827205.313355:0:14753:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a43480 - object 0x280000401:122 offset 0<>1408 00000100:00100000:2.0:1787827205.313402:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1787827205.313412:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280603520:192.168.203.124@tcp:4:cp.0 00010000:00010000:2.0:1787827205.313482:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000799c7467/0x268e1a602cc1d4e lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d92cc expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.313525:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a383c0 - object 0x280000400:137 offset 0<>4915 00000080:00200000:2.0:1787827205.313560:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x31:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.313573:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x32:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.313590:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.313602:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x32:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.313612:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x33:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.313617:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.313627:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x33:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.313639:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x34:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.313645:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.313657:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x34:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.313673:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x35:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.313680:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.313694:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x35:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.313704:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x36:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.313710:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.313723:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x36:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.313734:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x37:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.313741:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.313753:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x37:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.313770:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x38:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.313774:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.313789:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x38:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.313805:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x39:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.313812:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.313826:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x39:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.313837:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.313845:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.313857:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.313865:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.313869:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.313875:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.313920:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.313932:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.313944:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.313955:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.313965:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.313975:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.313984:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.313993:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1787827205.314001:0:8:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000003a0d2f85 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000067c86a9d/0x268e1a602cc1db7 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d93dd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.314011:0:8:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000003a0d2f85 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000067c86a9d/0x268e1a602cc1db7 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d93dd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.314035:0:8:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000067c86a9d/0x268e1a602cc1db7 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d93dd expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.314061:0:8:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a3a580 - object 0x2c0000400:139 offset 0<>838 00000080:00200000:2.0:1787827205.314082:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314110:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x3f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314115:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1787827205.314122:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a3a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280603776:192.168.203.124@tcp:4:cp.0 00000080:00200000:2.0:1787827205.314124:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x3f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314136:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x40:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314140:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314145:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x40:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314152:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x41:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314156:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314162:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x41:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314168:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x42:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314171:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314175:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x42:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314181:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x43:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314183:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314189:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x43:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314201:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x44:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314206:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314216:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x44:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314225:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x45:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314230:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314238:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x45:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314245:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x46:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314248:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314258:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x46:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314265:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x47:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314268:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314277:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x47:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314284:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x48:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314289:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314296:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x48:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314302:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x49:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314305:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314310:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x49:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314315:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314318:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314323:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314327:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314329:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314334:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314339:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314342:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314350:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314361:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314366:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314373:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314378:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x4e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314381:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314387:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x4e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314395:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x50:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314398:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314402:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x50:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314410:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x240001b71:0x51:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314412:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314420:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x240001b71:0x51:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314559:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314564:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314573:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314578:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314581:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314585:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314589:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0xd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314592:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314595:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0xd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314599:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0xb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314603:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314610:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0xb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314616:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314620:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314627:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314634:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x16:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314638:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.314645:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x16:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.314652:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x1f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.314657:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1787827205.317120:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000049806261 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005db90aac/0x268e1a602cc16d3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x79:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7c13 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.317203:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000049806261 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005db90aac/0x268e1a602cc16d3 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x79:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7c13 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827205.317248:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a39680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280602752:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.317620:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000048082e40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d935c527/0x268e1a602cc1480 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7608 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.317646:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000048082e40 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000d935c527/0x268e1a602cc1480 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7608 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827205.317673:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280602240:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.317712:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000004f7cf2d0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc14c6 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d76be expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.317744:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000004f7cf2d0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000cd0d19da/0x268e1a602cc14c6 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d76be expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827205.317773:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a39e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280601728:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.317834:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000f02bcfcf ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000066b0e2f9/0x268e1a602cc1647 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->163839) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7aa7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.318007:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000f02bcfcf ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000066b0e2f9/0x268e1a602cc1647 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->163839) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7aa7 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827205.318122:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280601600:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.318193:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000cf32beca ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042e2d893/0x268e1a602cc1877 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8057 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.318208:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000cf32beca ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042e2d893/0x268e1a602cc1877 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8057 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.318361:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042e2d893/0x268e1a602cc1877 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8057 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.318511:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aad680 - object 0x280000401:127 offset 0<>215293 00000008:00010000:0.0:1787827205.318555:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000b7b3b942 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1fc4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9932 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.318573:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000b7b3b942 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1fc4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9932 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1787827205.318584:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aad680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280604032:192.168.203.124@tcp:4:cp.0 00010000:00010000:0.0:1787827205.318604:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1fc4 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9932 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1787827205.318608:0:10954:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000be27f674 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dbb6ce8b/0x268e1a602cc1719 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7cc9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1787827205.318663:0:10954:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000be27f674 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000dbb6ce8b/0x268e1a602cc1719 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7cc9 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.318686:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aae580 - object 0x280000400:146 offset 0<>7187 00000100:00100000:3.0:1787827205.318688:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280603520:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.318726:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000007c50cbd4 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000067802747/0x268e1a602cc1baa lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8e88 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.318768:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000007c50cbd4 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000067802747/0x268e1a602cc1baa lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8e88 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.318775:0:10954:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000c76da257 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000baa403a3/0x268e1a602cc19b2 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d838a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827205.318803:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000067802747/0x268e1a602cc1baa lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8e88 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1787827205.318806:0:10954:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000c76da257 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000baa403a3/0x268e1a602cc19b2 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d838a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1787827205.318850:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a3a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280601856:192.168.203.124@tcp:4:cp.0 00000008:00100000:0.0:1787827205.318873:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aaf840 - object 0x280000400:131 offset 0<>1735 00000008:00010000:0.0:1787827205.318936:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000557b5d78 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000858fa2ee/0x268e1a602cc18bd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d810d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.318954:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000557b5d78 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000858fa2ee/0x268e1a602cc18bd lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d810d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.319018:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000858fa2ee/0x268e1a602cc18bd lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d810d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.319080:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aac3c0 - object 0x280000401:128 offset 0<>10934 00000008:00010000:0.0:1787827205.319131:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000837e6bce ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002b6e7c51/0x268e1a602cc200a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d99e8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.319148:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000837e6bce ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002b6e7c51/0x268e1a602cc200a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d99e8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787827205.319161:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aac3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280604416:192.168.203.124@tcp:4:cp.0 00010000:00010000:0.0:1787827205.319187:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002b6e7c51/0x268e1a602cc200a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d99e8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.319240:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aafc00 - object 0x280000400:147 offset 0<>1790 00000100:00100000:3.0:1787827205.319270:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:0.0:1787827205.319276:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000031f79b88 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000793fa4d7/0x268e1a602cc1bf0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8f3e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787827205.319285:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aafc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280604544:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.319305:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000031f79b88 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000793fa4d7/0x268e1a602cc1bf0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8f3e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.319363:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000793fa4d7/0x268e1a602cc1bf0 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8f3e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.319410:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aac000 - object 0x280000400:132 offset 0<>38517 00000008:00010000:0.0:1787827205.319437:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000008190f80b ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002466f344/0x268e1a602cc2050 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9a9e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.319454:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000008190f80b ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002466f344/0x268e1a602cc2050 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9a9e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.319483:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002466f344/0x268e1a602cc2050 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9a9e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.319546:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aad2c0 - object 0x280000400:148 offset 0<>6430 00000008:00010000:0.0:1787827205.319599:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000099716423 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d564540/0x268e1a602cc168d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7b5d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.319612:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000099716423 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d564540/0x268e1a602cc168d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7b5d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.319650:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d564540/0x268e1a602cc168d lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7b5d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.319693:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aacb40 - object 0x280000401:120 offset 0<>423 00000008:00010000:0.0:1787827205.319737:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000000c6046d ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000006426e69/0x268e1a602cc173c lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7d24 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1787827205.319746:0:10955:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:0.0:1787827205.319751:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000000c6046d ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000006426e69/0x268e1a602cc173c lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7d24 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1787827205.319758:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aacb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280604928:192.168.203.124@tcp:4:cp.0 00010000:00010000:0.0:1787827205.319802:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000006426e69/0x268e1a602cc173c lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7d24 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.319841:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aade00 - object 0x2c0000401:123 offset 0<>855 00000008:00010000:0.0:1787827205.319876:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000082650146 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e15a8114/0x268e1a602cc1fa1 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d98d7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.319924:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000082650146 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e15a8114/0x268e1a602cc1fa1 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d98d7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787827205.319929:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787827205.319949:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0aade00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280605056:192.168.203.124@tcp:4:cp.0 00010000:00010000:0.0:1787827205.319986:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e15a8114/0x268e1a602cc1fa1 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d98d7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.320025:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aae1c0 - object 0x2c0000400:146 offset 0<>4150 00000008:00010000:0.0:1787827205.320047:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000080093095 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000002c38b0e/0x268e1a602cc1aed lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d86ee expref: -99 pid: 48431 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1787827205.320061:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000080093095 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000002c38b0e/0x268e1a602cc1aed lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d86ee expref: -99 pid: 48431 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1787827205.320572:0:10955:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8ab3c584d000: committing for last_committed 4294967439 gen 1 00000100:00100000:3.0:1787827205.320610:0:10955:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8ab3d0aaf480 x1874671280602368/t4294967440(4294967440) o4->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 5000/448 e 0 to 0 dl 1787827221 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1787827205.320689:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000c448afe1 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000001c18fd01/0x268e1a602cc1b87 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->393215) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8e2d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827205.320710:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000002c38b0e/0x268e1a602cc1aed lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d86ee expref: -99 pid: 48431 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.321130:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000c448afe1 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000001c18fd01/0x268e1a602cc1b87 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->393215) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8e2d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1787827205.321157:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a3bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280603008:192.168.203.124@tcp:4:cp.0 00000008:00100000:0.0:1787827205.321159:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aacf00 - object 0x2c0000401:137 offset 0<>1048576 00000008:00010000:0.0:1787827205.321205:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000bcfb06bf ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000567b7b9e/0x268e1a602cc15de lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x76:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->36863) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7996 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.321221:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000bcfb06bf ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000567b7b9e/0x268e1a602cc15de lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x76:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->36863) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7996 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.321274:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000567b7b9e/0x268e1a602cc15de lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x76:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->36863) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7996 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.321317:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000099716423 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d564540/0x268e1a602cc168d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7b5d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.321335:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0aaed00 - object 0x2c0000401:118 offset 0<>35427 00000008:00010000:2.0:1787827205.321343:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000099716423 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006d564540/0x268e1a602cc168d lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7b5d expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.321372:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aacb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280604928:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.321375:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000418887c0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c6f7ebbd/0x268e1a602cc1fe7 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d998d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.321410:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000418887c0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c6f7ebbd/0x268e1a602cc1fe7 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d998d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.321442:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000c6f7ebbd/0x268e1a602cc1fe7 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d998d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.321502:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3f075ed00 - object 0x2c0000400:147 offset 0<>8441 00000008:00010000:0.0:1787827205.321532:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000cd66069f ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc1b33 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8d4d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1787827205.321541:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280605568:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.321560:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000cd66069f ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc1b33 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8d4d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.321594:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e1a14674/0x268e1a602cc1b33 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8d4d expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.321633:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3f075da40 - object 0x2c0000400:130 offset 0<>903 00000008:00010000:0.0:1787827205.321654:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000005d461592 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000037fa3f21/0x268e1a602cc202d lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9a43 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.321669:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000005d461592 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000037fa3f21/0x268e1a602cc202d lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9a43 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.321705:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000037fa3f21/0x268e1a602cc202d lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9a43 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.321741:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3f075cb40 - object 0x2c0000400:148 offset 0<>1694 00000008:00010000:0.0:1787827205.321763:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000001e9e2590 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000002fe1ebc5/0x268e1a602cc17c8 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7e90 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.321776:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000001e9e2590 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000002fe1ebc5/0x268e1a602cc17c8 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7e90 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.321812:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000002fe1ebc5/0x268e1a602cc17c8 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7e90 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.321848:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3f075e580 - object 0x2c0000401:125 offset 0<>6821 00000008:00010000:0.0:1787827205.321928:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000dc893b14 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc1c36 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8ff4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.321944:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000dc893b14 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc1c36 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8ff4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.321990:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc1c36 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8ff4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.322044:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3f075e1c0 - object 0x280000400:133 offset 0<>15476 00000008:00010000:0.0:1787827205.322071:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000e4c0e685 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009124e108/0x268e1a602cc20dc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9c0a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.322086:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000e4c0e685 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009124e108/0x268e1a602cc20dc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9c0a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.322146:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009124e108/0x268e1a602cc20dc lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9c0a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.322198:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3f075d680 - object 0x280000400:150 offset 0<>13879 00000100:00100000:0.0:1787827205.322257:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280606208:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.322810:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000b7b3b942 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1fc4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9932 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.322833:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000b7b3b942 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000005fed46c4/0x268e1a602cc1fc4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9932 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.322854:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aae580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280604160:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.322872:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000ffa5bab4 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000004b6759da/0x268e1a602cc2122 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->163839) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9cc0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.322914:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000ffa5bab4 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000004b6759da/0x268e1a602cc2122 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->163839) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9cc0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.322991:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000004b6759da/0x268e1a602cc2122 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->163839) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9cc0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.323126:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a38780 - object 0x280000400:151 offset 0<>159776 00000008:00010000:2.0:1787827205.323172:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000008190f80b ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002466f344/0x268e1a602cc2050 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9a9e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.323188:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000008190f80b ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002466f344/0x268e1a602cc2050 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9a9e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.323205:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aad2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280604800:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.323217:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000005785d62c ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.323226:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000005785d62c ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.323249:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000093fb86e0/0x268e1a602cc2168 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9d76 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.323278:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a3ad00 - object 0x280000400:152 offset 0<>423 00000008:00010000:2.0:1787827205.323342:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000007c50cbd4 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000067802747/0x268e1a602cc1baa lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8e88 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.323359:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000007c50cbd4 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000067802747/0x268e1a602cc1baa lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8e88 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.323383:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aaf840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280604288:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.323397:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000fe948028 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.323409:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000fe948028 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.323429:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.323464:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a38f00 - object 0x280000400:153 offset 0<>1659 00000100:00100000:2.0:1787827205.323497:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1787827205.323515:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a38f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280606592:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.325386:0:10954:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000dc893b14 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc1c36 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8ff4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.325425:0:10954:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000dc893b14 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000000da89d63/0x268e1a602cc1c36 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8ff4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.325452:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280606080:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.325472:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000077778719 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000fe5cc90c/0x268e1a602cc17eb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7eeb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.325484:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000077778719 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000fe5cc90c/0x268e1a602cc17eb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7eeb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.325577:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000fe5cc90c/0x268e1a602cc17eb lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7eeb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.325676:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a3b840 - object 0x280000401:125 offset 0<>123804 00000100:00100000:2.0:1787827205.325713:0:10956:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:2.0:1787827205.325725:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a3b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280606848:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.325849:0:10954:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000001e9e2590 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000002fe1ebc5/0x268e1a602cc17c8 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7e90 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.325874:0:10954:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000001e9e2590 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000002fe1ebc5/0x268e1a602cc17c8 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7e90 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.325929:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280605952:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.325966:0:10954:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000000c6046d ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000006426e69/0x268e1a602cc173c lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7d24 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.325995:0:10954:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000000c6046d ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000006426e69/0x268e1a602cc173c lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7d24 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.326022:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aade00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280605056:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.326048:0:10954:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000837e6bce ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002b6e7c51/0x268e1a602cc200a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d99e8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.326063:0:10954:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000837e6bce ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000002b6e7c51/0x268e1a602cc200a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d99e8 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.326077:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aafc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280604544:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.326108:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000019200feb ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000019e1f18d/0x268e1a602cc1dda lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9438 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.326123:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000019200feb ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000019e1f18d/0x268e1a602cc1dda lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9438 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.326159:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000019e1f18d/0x268e1a602cc1dda lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9438 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.326212:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a3b480 - object 0x280000400:139 offset 0<>2858 00000100:00100000:2.0:1787827205.326379:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ffc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280606976:192.168.203.124@tcp:4:cp.0 00000100:00100000:2.0:1787827205.326430:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280607488:192.168.203.124@tcp:4:cp.0 00000100:00100000:3.0:1787827205.326537:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280607232:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.326733:0:10954:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000fe948028 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.326761:0:10954:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000fe948028 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000788330bc/0x268e1a602cc21ae lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9e2c expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.326797:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a38f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280606592:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.326828:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000086a033fb ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e2b0e04d/0x268e1a602cc19d5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1204223) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d83e5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.326842:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000086a033fb ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e2b0e04d/0x268e1a602cc19d5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1204223) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d83e5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787827205.327103:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280607744:192.168.203.124@tcp:4:cp.0 00010000:00010000:2.0:1787827205.327560:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e2b0e04d/0x268e1a602cc19d5 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1204223) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d83e5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.328283:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a38000 - object 0x280000401:132 offset 0<>1200870 00000008:00010000:2.0:1787827205.328324:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.328335:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.328413:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.328505:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a38b40 - object 0x280000400:157 offset 0<>123804 00000008:00010000:2.0:1787827205.328539:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000f3a11fe9 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000044ebe087/0x268e1a602cc1e20 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d94ee expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.328559:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000f3a11fe9 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000044ebe087/0x268e1a602cc1e20 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d94ee expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.328594:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000044ebe087/0x268e1a602cc1e20 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d94ee expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.328633:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a39a40 - object 0x280000400:140 offset 0<>4218 00000008:00010000:2.0:1787827205.328648:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.328657:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.328754:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1787827205.328881:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000e4c0e685 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009124e108/0x268e1a602cc20dc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9c0a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.328913:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a3b0c0 - object 0x280000400:158 offset 0<>170595 00000100:00100000:2.0:1787827205.328955:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280607872:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827205.329016:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000e4c0e685 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000009124e108/0x268e1a602cc20dc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9c0a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.329052:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a3b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280608384:192.168.203.124@tcp:4:cp.0 00000100:00100000:2.0:1787827205.329115:0:10957:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:3.0:1787827205.329125:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280606208:192.168.203.124@tcp:4:cp.0 00000100:00100000:2.0:1787827205.329127:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a39a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280608256:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827205.329258:0:14753:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000079efc04b ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000003a3730c8/0x268e1a602cc1e66 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d95a4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.329273:0:14753:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000079efc04b ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000003a3730c8/0x268e1a602cc1e66 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d95a4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.329836:0:14753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000003a3730c8/0x268e1a602cc1e66 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d95a4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.329877:0:14753:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a41a40 - object 0x280000400:141 offset 0<>2192 00000008:00010000:3.0:1787827205.330284:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000082650146 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e15a8114/0x268e1a602cc1fa1 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d98d7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1787827205.330311:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000082650146 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e15a8114/0x268e1a602cc1fa1 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d98d7 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1787827205.330339:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aae1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280605184:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827205.330356:0:14753:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000006031f8f3 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000076a9a6af/0x268e1a602cc143a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7552 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.330411:0:14753:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000006031f8f3 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000076a9a6af/0x268e1a602cc143a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7552 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.330439:0:14753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000076a9a6af/0x268e1a602cc143a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7552 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.330595:0:14753:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a41e00 - object 0x2c0000401:112 offset 0<>269 00000100:00100000:2.0:1787827205.330660:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280608640:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827205.330675:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000031f79b88 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000793fa4d7/0x268e1a602cc1bf0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8f3e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1787827205.330752:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000031f79b88 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000793fa4d7/0x268e1a602cc1bf0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8f3e expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1787827205.330784:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280604672:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827205.330838:0:14753:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000d8ccb11d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006ec07da3/0x268e1a602cc1a1b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d849b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.330854:0:14753:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000d8ccb11d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006ec07da3/0x268e1a602cc1a1b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d849b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.330880:0:14753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006ec07da3/0x268e1a602cc1a1b lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d849b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.330931:0:14753:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a40b40 - object 0x280000401:133 offset 0<>2475 00000100:00100000:3.0:1787827205.330961:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280608768:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.332770:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000019200feb ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000019e1f18d/0x268e1a602cc1dda lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9438 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.332805:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000019200feb ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000019e1f18d/0x268e1a602cc1dda lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9438 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.332853:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a3b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280607360:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.332941:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000ffa5bab4 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000004b6759da/0x268e1a602cc2122 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->163839) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9cc0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.333079:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000ffa5bab4 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000004b6759da/0x268e1a602cc2122 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->163839) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9cc0 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.333122:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a38780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280606336:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.333172:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000030b32ee3 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a1c64be4/0x268e1a602cc1eac lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->57343) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d965a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.333201:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000030b32ee3 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a1c64be4/0x268e1a602cc1eac lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->57343) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d965a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.333293:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a1c64be4/0x268e1a602cc1eac lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->57343) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d965a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.333379:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a39680 - object 0x280000400:142 offset 0<>55540 00000008:00010000:2.0:1787827205.333404:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000b70849e7 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000088bc067/0x268e1a602cc2424 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da492 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.333419:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000b70849e7 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000088bc067/0x268e1a602cc2424 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da492 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787827205.333444:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280609024:192.168.203.124@tcp:4:cp.0 00010000:00010000:2.0:1787827205.333582:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000088bc067/0x268e1a602cc2424 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0xa2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da492 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.333718:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a39e00 - object 0x280000400:162 offset 0<>255050 00000100:00100000:2.0:1787827205.333766:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a39e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280609152:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.333856:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000a485f1d1 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000609cf18d/0x268e1a602cc20b9 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->36863) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9baf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.333912:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000a485f1d1 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000609cf18d/0x268e1a602cc20b9 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->36863) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9baf expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.333946:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280607104:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.333981:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000c785b8c8 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f3067bab/0x268e1a602cc189a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->200703) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d80b2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.333989:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000245f19da ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.333994:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000c785b8c8 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f3067bab/0x268e1a602cc189a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->200703) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d80b2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.334115:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000245f19da ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a2af288a/0x268e1a602cc218b lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9dd1 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827205.334167:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f3067bab/0x268e1a602cc189a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->200703) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d80b2 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.334333:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a3bc00 - object 0x2c0000401:128 offset 0<>199288 00000008:00010000:0.0:1787827205.334337:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000080093095 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000002c38b0e/0x268e1a602cc1aed lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d86ee expref: -99 pid: 48431 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1787827205.334340:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280607744:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.334356:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000f4293618 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.334400:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000f4293618 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.334410:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000077778719 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000fe5cc90c/0x268e1a602cc17eb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7eeb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827205.334443:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.334494:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a3a580 - object 0x2c0000400:157 offset 0<>6821 00000008:00010000:2.0:1787827205.334600:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000076a44bc9 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000004baa755b/0x268e1a602cc1c9f lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9105 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.334629:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000076a44bc9 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000004baa755b/0x268e1a602cc1c9f lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9105 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.334665:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280608000:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.334714:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000025585445 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009580493c/0x268e1a602cc1c59 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d904f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.334739:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000025585445 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000009580493c/0x268e1a602cc1c59 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d904f expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.334764:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280607616:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.334790:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000b0c4c367 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000008a98f536/0x268e1a602cc166a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7b02 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.334802:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000b0c4c367 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000008a98f536/0x268e1a602cc166a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7b02 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.334874:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000008a98f536/0x268e1a602cc166a lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7b02 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.334960:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a3a940 - object 0x2c0000401:120 offset 0<>37842 00000008:00010000:2.0:1787827205.334997:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000005b104820 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.335006:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000005b104820 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.335063:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000080093095 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000002c38b0e/0x268e1a602cc1aed lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x772721ad1f0d86ee expref: -99 pid: 48431 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.335068:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e152a059/0x268e1a602cc232f lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da215 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827205.335125:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0aacf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280605312:192.168.203.124@tcp:4:dd.0 00000100:00100000:0.0:1787827205.335144:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a3a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280609408:192.168.203.124@tcp:4:cp.0 00000008:00100000:2.0:1787827205.335157:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a3a1c0 - object 0x2c0000400:159 offset 0<>31711 00000008:00010000:3.0:1787827205.335182:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000077778719 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000fe5cc90c/0x268e1a602cc17eb lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7eeb expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.335191:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000007480b0d0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ad77f645/0x268e1a602cc1d71 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9327 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.335208:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000007480b0d0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ad77f645/0x268e1a602cc1d71 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9327 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787827205.335218:0:10957:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:0.0:1787827205.335230:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a3bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280609280:192.168.203.124@tcp:4:cp.0 00010000:00010000:2.0:1787827205.335242:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ad77f645/0x268e1a602cc1d71 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x8a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9327 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.335313:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d08c4b40 - object 0x2c0000400:138 offset 0<>1613 00000100:00100000:2.0:1787827205.335362:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a3a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280609664:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.335383:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000f3a11fe9 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000044ebe087/0x268e1a602cc1e20 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d94ee expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1787827205.335393:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a3b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280606848:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.335418:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000f3a11fe9 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000044ebe087/0x268e1a602cc1e20 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d94ee expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.335422:0:10955:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1787827205.335438:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a3a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280609536:192.168.203.124@tcp:4:cp.0 00000100:00100000:0.0:1787827205.335445:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a39a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280608256:192.168.203.124@tcp:4:cp.0 00000100:00100000:0.0:1787827205.335458:0:10957:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:0.0:1787827205.335472:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08c4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280609792:192.168.203.124@tcp:4:cp.0 00000008:00010000:3.0:1787827205.335547:0:14753:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000708fd48e ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a90e06a9/0x268e1a602cc246a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->438271) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da548 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.335565:0:14753:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000708fd48e ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a90e06a9/0x268e1a602cc246a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->438271) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da548 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.335630:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000006031f8f3 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000076a9a6af/0x268e1a602cc143a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7552 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.335657:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000006031f8f3 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000076a9a6af/0x268e1a602cc143a lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d7552 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.335685:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280608640:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.335732:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000001050ebda ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000056ea50f1/0x268e1a602cc23bb lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0xa1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->229375) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da381 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.335742:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000001050ebda ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000056ea50f1/0x268e1a602cc23bb lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0xa1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->229375) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da381 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.335985:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 0000000056ea50f1/0x268e1a602cc23bb lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0xa1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->229375) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da381 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.335990:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.336156:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000009b8f4015 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000ce919349/0x268e1a602cc230c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da1ba expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827205.336173:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a3b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280608384:192.168.203.124@tcp:4:cp.0 00000008:00100000:2.0:1787827205.336177:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d08c4000 - object 0x2c0000400:161 offset 0<>227516 00000008:00010000:2.0:1787827205.336223:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000000e7c4f72 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000646bfc9f/0x268e1a602cc1dfd lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->20479) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9493 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.336237:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000000e7c4f72 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000646bfc9f/0x268e1a602cc1dfd lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->20479) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9493 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.336283:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000646bfc9f/0x268e1a602cc1dfd lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->20479) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9493 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1787827205.336291:0:14753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a90e06a9/0x268e1a602cc246a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0xa3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->438271) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da548 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.336336:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d08c61c0 - object 0x2c0000400:140 offset 0<>18269 00000008:00010000:2.0:1787827205.336423:0:10955:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.336456:0:10955:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000001b79965d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000042fad9e8/0x268e1a602cc22c6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->126975) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da104 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.336473:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a38b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280608128:192.168.203.124@tcp:4:cp.0 00000100:00100000:2.0:1787827205.336477:0:10955:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1787827205.336483:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08c61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280610048:192.168.203.124@tcp:4:cp.0 00000080:00200000:2.0:1787827205.336804:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x1f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.336825:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x21:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.336836:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.336852:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x21:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.336858:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x23:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.336861:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.336865:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x23:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.336870:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x25:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.336873:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827205.336876:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b72:0x25:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827205.336881:0:8:0:(vvp_io.c:1889:vvp_io_init()) [0x200001b72:0x27:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827205.336906:0:8:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1787827205.336981:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000d8ccb11d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006ec07da3/0x268e1a602cc1a1b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d849b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.336997:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000d8ccb11d ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006ec07da3/0x268e1a602cc1a1b lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d849b expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827205.337010:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280608768:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.337032:0:10956:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000816484c2 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc1c13 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8f99 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.337058:0:10956:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000816484c2 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f67370d2/0x268e1a602cc1c13 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8f99 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827205.337080:0:10956:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280607232:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.337123:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000b919caee ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e045a7fc/0x268e1a602cc1e43 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9549 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.337134:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000b919caee ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e045a7fc/0x268e1a602cc1e43 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9549 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.337178:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e045a7fc/0x268e1a602cc1e43 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9549 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.337243:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3f075fc00 - object 0x2c0000400:141 offset 0<>3006 00000100:00100000:0.0:1787827205.337312:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280610304:192.168.203.124@tcp:4:cp.0 00000008:00100000:3.0:1787827205.337312:0:14753:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a41680 - object 0x280000400:163 offset 0<>434581 00000008:00010000:3.0:1787827205.337372:0:14753:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000bb6dca3c ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c5f94666/0x268e1a602cc1ef2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9710 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.337393:0:14753:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000bb6dca3c ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c5f94666/0x268e1a602cc1ef2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9710 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.337445:0:14753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000c5f94666/0x268e1a602cc1ef2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9710 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827205.337454:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280610176:192.168.203.124@tcp:4:cp.0 00000008:00100000:3.0:1787827205.337500:0:14753:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a40780 - object 0x280000400:143 offset 0<>6213 00000008:00010000:3.0:1787827205.337533:0:14753:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000033e67cc3 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006476617f/0x268e1a602cc24b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1204223) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da5fe expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.337826:0:14753:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000033e67cc3 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006476617f/0x268e1a602cc24b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1204223) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da5fe expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827205.338432:0:14753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000006476617f/0x268e1a602cc24b0 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1204223) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da5fe expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.339068:0:14753:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a42d00 - object 0x280000400:164 offset 0<>1200870 00000008:00010000:3.0:1787827205.339115:0:14753:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000001edf14ef ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000077ce8885/0x268e1a602cc1a61 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8551 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.339128:0:14753:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000001edf14ef ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000077ce8885/0x268e1a602cc1a61 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8551 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1787827205.339157:0:10955:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00010000:00010000:3.0:1787827205.339161:0:14753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000077ce8885/0x268e1a602cc1a61 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8551 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.339171:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a42d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280610560:192.168.203.124@tcp:4:cp.0 00000008:00100000:3.0:1787827205.339199:0:14753:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a42940 - object 0x280000401:134 offset 0<>1611 00000008:00010000:3.0:1787827205.339222:0:14753:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000000a62de75 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000da47bf4e/0x268e1a602cc24f6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da6b4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787827205.339233:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:3.0:1787827205.339236:0:14753:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000000a62de75 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000da47bf4e/0x268e1a602cc24f6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da6b4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787827205.339257:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a42940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280610688:192.168.203.124@tcp:4:cp.0 00010000:00010000:3.0:1787827205.339269:0:14753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000da47bf4e/0x268e1a602cc24f6 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da6b4 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.339303:0:14753:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a42580 - object 0x280000400:165 offset 0<>2475 00000008:00010000:3.0:1787827205.339339:0:14753:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000aabde55f ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d59ca25b/0x268e1a602cc1f38 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d97c6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1787827205.339350:0:14753:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000aabde55f ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d59ca25b/0x268e1a602cc1f38 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d97c6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1787827205.339383:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a42580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280610816:192.168.203.124@tcp:4:cp.0 00010000:00010000:3.0:1787827205.339431:0:14753:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d59ca25b/0x268e1a602cc1f38 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d97c6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1787827205.339491:0:14753:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d0a40f00 - object 0x280000400:144 offset 0<>234 00000100:00100000:0.0:1787827205.339539:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280610944:192.168.203.124@tcp:4:cp.0 00000100:00100000:2.0:1787827205.339928:0:10957:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8ab3c584d000: committing for last_committed 4294967460 gen 1 00000100:00100000:2.0:1787827205.340036:0:10957:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8ab3d0a41e00 x1874671280608640/t4294967461(4294967461) o4->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 760/448 e 0 to 0 dl 1787827221 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1787827205.340063:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000007480b0d0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ad77f645/0x268e1a602cc1d71 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9327 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.340102:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000007480b0d0 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ad77f645/0x268e1a602cc1d71 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d9327 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.340128:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d08c4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280609792:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.340155:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000f4293618 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.340175:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000f4293618 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000e83ae507/0x268e1a602cc22a3 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da0a9 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.340195:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a3a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280609408:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.340252:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000086a033fb ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e2b0e04d/0x268e1a602cc19d5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1204223) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d83e5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:2.0:1787827205.340472:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000086a033fb ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000e2b0e04d/0x268e1a602cc19d5 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1204223) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d83e5 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1787827205.340487:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a38000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280607872:192.168.203.124@tcp:4:cp.0 00000008:00010000:2.0:1787827205.340513:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000003d236ab8 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ae88d61a/0x268e1a602cc248d lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da5a3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.340527:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000003d236ab8 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ae88d61a/0x268e1a602cc248d lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da5a3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.340573:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000ae88d61a/0x268e1a602cc248d lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da5a3 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.340633:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d08c5680 - object 0x2c0000400:164 offset 0<>26716 00000008:00010000:2.0:1787827205.340661:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000010e12097 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000deaebfaf/0x268e1a602cc1e89 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d95ff expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.340673:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000010e12097 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000deaebfaf/0x268e1a602cc1e89 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d95ff expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.340703:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000deaebfaf/0x268e1a602cc1e89 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d95ff expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.340746:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d08c7c00 - object 0x2c0000400:142 offset 0<>938 00000008:00010000:2.0:1787827205.340771:0:48406:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827205.340789:0:48406:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000081389d49 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827205.340876:0:48406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a7a14bd9/0x268e1a602cc2582 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0da820 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827205.341066:0:48406:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3d08c5a40 - object 0x280000400:167 offset 0<>317548 00000100:00100000:2.0:1787827205.341129:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08c5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280611328:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.341163:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000aabde55f ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d59ca25b/0x268e1a602cc1f38 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d97c6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.341189:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000aabde55f ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000d59ca25b/0x268e1a602cc1f38 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d97c6 expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827205.341209:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280610944:192.168.203.124@tcp:4:cp.0 00000100:00100000:0.0:1787827205.341221:0:10957:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8ab3c584d000: committing for last_committed 4294967462 gen 1 00000100:00100000:0.0:1787827205.341241:0:10957:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8ab3d08c4b40 x1874671280609792/t4294967463(4294967463) o4->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 2104/448 e 0 to 0 dl 1787827221 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1787827205.341263:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000069e44aa ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000007c03f664/0x268e1a602cc1aa7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8607 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.341277:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000069e44aa ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000007c03f664/0x268e1a602cc1aa7 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8607 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.341352:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 000000007c03f664/0x268e1a602cc1aa7 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->319487) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d8607 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.341482:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3f075d2c0 - object 0x280000401:135 offset 0<>317548 00000100:00100000:0.0:1787827205.341514:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280611456:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.341683:0:10954:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000030b32ee3 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a1c64be4/0x268e1a602cc1eac lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->57343) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d965a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.341713:0:10954:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000030b32ee3 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 00000000a1c64be4/0x268e1a602cc1eac lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->57343) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d965a expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827205.341738:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a39680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280608896:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.341765:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000007ba4ef22 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000022463a61/0x268e1a602cc1903 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d81c3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.341776:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000007ba4ef22 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000022463a61/0x268e1a602cc1903 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d81c3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.341847:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000022463a61/0x268e1a602cc1903 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d81c3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.341969:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3f075c3c0 - object 0x280000401:129 offset 0<>159678 00000100:00100000:0.0:1787827205.342005:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280611584:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.342597:0:10957:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000f060cf43 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000007183923b/0x268e1a602cc1ce5 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d91bb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:0.0:1787827205.342629:0:10957:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000f060cf43 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 000000007183923b/0x268e1a602cc1ce5 lrc: 2/0,0 mode: PW/PW res: [0x2c0000400:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d91bb expref: -99 pid: 48442 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827205.342659:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280609024:192.168.203.124@tcp:4:cp.0 00000008:00010000:0.0:1787827205.342687:0:41:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 00000000289d3f60 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f76f7717/0x268e1a602cc196c lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d82d4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:0.0:1787827205.342701:0:41:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000289d3f60 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f76f7717/0x268e1a602cc196c lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d82d4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827205.342757:0:41:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f76f7717/0x268e1a602cc196c lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x772721ad1f0d82d4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:0.0:1787827205.342811:0:41:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8ab3f075c780 - object 0x2c0000401:131 offset 0<>42517 00000100:00100000:3.0:1787827205.342869:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280611712:192.168.203.124@tcp:4:cp.0 00010000:00010000:0.0:1787827205.990070:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827207.014146:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08c52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10956:1874671280611840:192.168.203.124@tcp:400:kworker.0 00000100:00100000:3.0:1787827207.014218:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08c4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280612224:192.168.203.124@tcp:400:kworker.0 00000100:00100000:2.0:1787827207.014239:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08c6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280612096:192.168.203.124@tcp:400:kworker.0 00000100:00100000:0.0:1787827207.014274:0:10956:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08c6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:27aedc06-3022-4c11-aafb-6bd27e69b836:10956:1874671280612352:192.168.203.124@tcp:400:kworker.0 00010000:00010000:0.0:1787827208.038048:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827210.086103:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827212.134060:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827212.134137:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08c4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10957:1874671280612480:192.168.203.124@tcp:400:kworker.0 00000100:00100000:3.0:1787827212.134167:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08c7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280612736:192.168.203.124@tcp:400:kworker.0 00000100:00100000:0.0:1787827212.134261:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08c5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280612992:192.168.203.124@tcp:400:kworker.0 00010000:00010000:3.0:1787827214.182050:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827216.230006:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827217.254144:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d08c6940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10954:1874671280613120:192.168.203.124@tcp:400:kworker.0 00000100:00100000:3.0:1787827217.254160:0:10955:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280613248:192.168.203.124@tcp:400:kworker.0 00000100:00100000:2.0:1787827217.254228:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280613504:192.168.203.124@tcp:400:kworker.0 00000100:00100000:0.0:1787827217.254261:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280613632:192.168.203.124@tcp:400:kworker.0 00010000:00010000:2.0:1787827218.278118:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827220.326044:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1787827221.350055:0:10955:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827205/real 0] req@ffff8ab3f075c780 x1874671280611712/t0(0) o4->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 488/448 e 0 to 1 dl 1787827221 ref 3 fl Rpc:Xr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00000400:3.0:1787827221.350055:0:10954:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827205/real 0] req@ffff8ab3f075c3c0 x1874671280611584/t0(0) o4->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 488/448 e 0 to 1 dl 1787827221 ref 3 fl Rpc:Xr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1787827221.350141:0:10955:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:3.0:1787827221.350142:0:10954:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1787827221.350149:0:10955:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff8ab3c584d000: Connection to lustre-OST0001 (at 192.168.203.124@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:02000400:3.0:1787827221.350152:0:10954:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff8ab3c584d000: Connection to lustre-OST0000 (at 192.168.203.124@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1787827221.350162:0:10954:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00000400:3.0:1787827221.350191:0:10954:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8ab3d08c7c00 x1874671280611200/t0(0) o4->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 1432/448 e 0 to 1 dl 1787827221 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1787827221.350288:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827221.350297:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827221.350301:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827221.350308:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1751 00000100:00080000:2.0:1787827221.350323:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827221.350335:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827221.350350:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280614144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827221.363904:0:10955:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:3.0:1787827221.363935:0:10954:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db4000 already not connected (conn 1, was 1): DISCONN 00000100:00080000:2.0:1787827221.363941:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827221.363948:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827221.363953:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000400:0.0:1787827221.363966:0:10955:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8ab3d0a42d00 x1874671280610560/t0(0) o4->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 488/448 e 0 to 1 dl 1787827221 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1787827221.363967:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1751 00000100:00000400:3.0:1787827221.363967:0:10954:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8ab3d08c5680 x1874671280611072/t0(0) o4->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 488/448 e 0 to 1 dl 1787827221 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1787827221.364006:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827221.364034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:0.0:1787827221.364077:0:10955:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db0800 already not connected (conn 2, was 1): CONNECTING 00000100:00080000:3.0:1787827221.364090:0:10954:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db4000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1787827221.364097:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280614272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000400:0.0:1787827221.364098:0:10955:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8ab3f075fc00 x1874671280610304/t0(0) o4->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 3496/448 e 0 to 1 dl 1787827221 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00000400:3.0:1787827221.364100:0:10954:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8ab3d0a42940 x1874671280610688/t0(0) o4->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 2104/448 e 0 to 1 dl 1787827221 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1787827221.364144:0:10954:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db0800 already not connected (conn 2, was 1): CONNECTING 00000100:00080000:0.0:1787827221.364153:0:10955:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db4000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:0.0:1787827221.364163:0:10955:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8ab3d08c61c0 x1874671280610048/t0(0) o4->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 488/448 e 0 to 1 dl 1787827221 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787827221.364173:0:10954:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:0.0:1787827221.364182:0:10955:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db4000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:0.0:1787827221.364191:0:10955:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8ab3d0a3a940 x1874671280609536/t0(0) o4->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 488/448 e 0 to 1 dl 1787827221 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787827221.364193:0:10954:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10954:1874671280613760:192.168.203.124@tcp:400:kworker.0 00000100:00100000:2.0:1787827221.364231:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280614144:192.168.203.124@tcp:8:kworker.0 00000100:00080000:0.0:1787827221.364236:0:10955:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db4000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:0.0:1787827221.364247:0:10955:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8ab3d0a3a1c0 x1874671280609664/t0(0) o4->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 488/448 e 0 to 1 dl 1787827221 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1787827221.364259:0:10955:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db4000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:0.0:1787827221.364264:0:10955:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8ab3d0a39e00 x1874671280609152/t0(0) o4->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 488/448 e 0 to 1 dl 1787827221 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1787827221.364275:0:10955:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db0800 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1787827221.364315:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280614272:192.168.203.124@tcp:8:kworker.0 00000100:00000400:3.0:1787827221.364354:0:10957:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8ab3f075d2c0 x1874671280611456/t0(0) o4->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 488/448 e 0 to 1 dl 1787827221 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1787827221.364374:0:10957:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db0800 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:3.0:1787827221.364382:0:10957:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8ab3d0a40780 x1874671280610432/t0(0) o4->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 6704/448 e 0 to 1 dl 1787827221 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1787827221.364394:0:10957:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db0800 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:3.0:1787827221.364400:0:10957:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8ab3d08c4000 x1874671280609920/t0(0) o4->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 488/448 e 0 to 1 dl 1787827221 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1787827221.364412:0:10957:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db4000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:3.0:1787827221.364417:0:10957:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8ab3d0a3bc00 x1874671280609280/t0(0) o4->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:6/4 lens 488/448 e 0 to 1 dl 1787827221 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1787827221.364425:0:10957:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db4000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:3.0:1787827221.364433:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280614016:192.168.203.124@tcp:400:kworker.0 00000100:00100000:3.0:1787827221.364457:0:10957:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:3.0:1787827221.364462:0:10957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280613888:192.168.203.124@tcp:400:kworker.0 00010000:00010000:3.0:1787827221.367815:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827222.374092:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1787827223.398056:0:10955:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827207/real 0] req@ffff8ab3d08c4780 x1874671280612224/t0(0) o400->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827223 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1787827223.398144:0:10957:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827207/real 0] req@ffff8ab3d08c43c0 x1874671280611968/t0(0) o400->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827223 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827223.398200:0:10957:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8ab3d08c43c0 x1874671280611968/t0(0) o400->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827223 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827223.398212:0:10957:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1787827223.398217:0:10957:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8ab3c584d000: Connection to lustre-MDT0000 (at 192.168.203.124@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1787827223.398232:0:10957:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00000400:3.0:1787827223.398300:0:10954:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827207/real 0] req@ffff8ab3d08c6580 x1874671280612096/t0(0) o400->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827223 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827223.398320:0:10954:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8ab3d08c6580 x1874671280612096/t0(0) o400->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827223 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827223.398328:0:10954:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1787827223.398332:0:10954:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff8ab3c584d000: Connection to lustre-MDT0001 (at 192.168.203.124@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1787827223.398336:0:10954:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1787827223.410938:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827223.410951:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827223.410961:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827223.410989:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1751 00000100:00080000:2.0:1787827223.411007:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827223.411045:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280614400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827223.411079:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827223.411083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827223.411086:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827223.411091:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1751 00000100:00080000:2.0:1787827223.411101:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827223.411125:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827223.411143:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280614400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827223.411151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280614528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827223.411170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827223.411174:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827223.411177:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827223.411182:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1751 00000100:00080000:2.0:1787827223.411189:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827223.411196:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827223.411211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280614656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827223.411230:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827223.411234:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827223.411238:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827223.411242:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827223.411256:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827223.411256:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280614528:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787827223.411269:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827223.411273:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827223.411277:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827223.411280:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827223.411284:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1787827223.411306:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280614656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827223.418564:0:10955:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8ab3d08c4780 x1874671280612224/t0(0) o400->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827223 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827223.418583:0:10955:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db0800 already not connected (conn 2, was 1): CONNECTING 00010000:00010000:2.0:1787827224.422061:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827226.470026:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827227.494062:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1787827228.518013:0:10954:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827212/real 0] req@ffff8ab3d08c7840 x1874671280612608/t0(0) o400->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827228 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1787827228.518013:0:10957:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827212/real 0] req@ffff8ab3d08c5e00 x1874671280612992/t0(0) o400->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827228 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1787827228.518015:0:10956:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827212/real 0] req@ffff8ab3d08c70c0 x1874671280612864/t0(0) o400->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827228 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827228.518081:0:10956:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8ab3d08c70c0 x1874671280612864/t0(0) o400->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827228 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827228.518087:0:10957:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8ab3d08c5e00 x1874671280612992/t0(0) o400->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827228 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827228.518118:0:10957:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db4000 already not connected (conn 2, was 1): CONNECTING 00000100:00080000:0.0:1787827228.518119:0:10956:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db0800 already not connected (conn 2, was 1): CONNECTING 00010000:00010000:0.0:1787827228.518179:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827228.528965:0:10954:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8ab3d08c7840 x1874671280612608/t0(0) o400->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827228 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827228.528975:0:10954:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8ab3c584e000 already not connected (conn 2, was 1): CONNECTING 00010000:00010000:0.0:1787827230.566094:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827231.590102:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827221/real 0] req@ffff8ab3f043a1c0 x1874671280614144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827231 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827231.590202:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280614144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827231 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827231.590210:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827221/real 0] req@ffff8ab3f0439680 x1874671280614272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827231 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827231.590220:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280614272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827231 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827233.637992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827233.637994:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827223/real 0] req@ffff8ab3f0439a40 x1874671280614528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827233 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1787827233.638002:0:10957:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827217/real 0] req@ffff8ab3f043b0c0 x1874671280613504/t0(0) o400->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827233 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827233.638005:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827233.638012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827233.638023:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827233.638031:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827233.638043:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1787827233.638044:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280614528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827233 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827233.638051:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827233.638064:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827233.638071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827233.638082:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1787827233.638083:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827223/real 0] req@ffff8ab3f04392c0 x1874671280614656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827233 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827233.638106:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280614656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827233 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827233.638124:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827223/real 0] req@ffff8ab3f043bc00 x1874671280614400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827233 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:2.0:1787827233.638128:0:10956:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827217/real 0] req@ffff8ab3f043b480 x1874671280613376/t0(0) o400->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827233 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827233.638163:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280614400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827233 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827233.638170:0:10956:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8ab3f043b480 x1874671280613376/t0(0) o400->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827233 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827233.638195:0:10956:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8ab3c584e800 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:0.0:1787827233.649987:0:10957:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8ab3f043b0c0 x1874671280613504/t0(0) o400->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827233 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827233.649996:0:10957:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8ab3d2db0800 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:0.0:1787827237.733983:0:10957:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827221/real 0] req@ffff8ab3f0438b40 x1874671280613888/t0(0) o400->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827237 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827237.743113:0:10957:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8ab3f0438b40 x1874671280613888/t0(0) o400->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827237 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827237.743128:0:10957:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8ab3c584e000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:0.0:1787827237.743136:0:10957:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1787827221/real 0] req@ffff8ab3f0438000 x1874671280614016/t0(0) o400->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827237 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827237.743150:0:10957:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8ab3f0438000 x1874671280614016/t0(0) o400->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827237 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827237.743157:0:10957:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8ab3c584e800 already not connected (conn 2, was 1): CONNECTING 00000100:00080000:2.0:1787827238.758012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827238.758027:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827238.758031:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827238.758036:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827238.758041:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827238.758048:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827238.758053:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827238.758057:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827238.758061:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827238.758066:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827243.878116:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827243.878138:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827243.878141:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827243.878144:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827243.878146:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827243.878149:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827243.878151:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827243.878153:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827243.878161:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827243.878163:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827248.998022:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827248.998034:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827248.998038:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827248.998042:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827248.998045:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827248.998048:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827248.998051:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827248.998055:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827248.998057:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827248.998060:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1787827253.094022:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827254.118089:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827254.118098:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827254.118101:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827254.118103:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827254.118105:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827254.118107:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827254.118109:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827254.118112:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827254.118114:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827254.118117:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1787827255.142040:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827257.190027:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827259.238003:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827259.238013:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827259.238018:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827259.238021:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827259.238023:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827259.238025:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827259.238029:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827259.238031:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827259.238033:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827259.238035:0:15659:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1787827259.238086:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827261.286081:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:1.1F:1787827262.118017:0:10946:0:(socklnd_cb.c:2294:ksocknal_find_timed_out_conn()) Timeout sending data to 12345-192.168.203.124@tcp (192.168.203.124:988) the network or that node may be down. 00000400:02000040:2.0:1787827262.245941:0:10951:0:(lib-move.c:3830:lnet_print_recovery_list()) 1 local NIs in recovery (showing 1): 192.168.203.24@tcp 00010000:00010000:0.0:1787827263.334031:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827264.358027:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827264.358035:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827264.358038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827264.358041:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827264.358044:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827264.358047:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827264.358050:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827264.358052:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827264.358055:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827264.358058:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000400:00000100:0.0:1787827265.190288:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.124:988 00000400:00000100:0.0:1787827265.190323:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827265.190338:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 3 len 152294 192.168.203.124@tcp->192.168.203.124@tcp 00000100:00020000:0.0:1787827265.190354:0:10943:0:(events.c:192:client_bulk_callback()) event type 1, status -113, req ffff8ab3d0a42d00 desc ffff8ab3f04bd800 mbits 1874671280610561 00000800:00000100:0.0:1787827265.195506:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000800:00000100:0.0:1787827265.195534:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000800:00000100:0.0:1787827265.195544:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000800:00000100:0.0:1787827265.195564:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00000400:2.0:1787827265.195569:0:10957:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827212/real 1787827265] req@ffff8ab3d08c4f00 x1874671280612480/t0(0) o400->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 224/224 e 0 to 1 dl 1787827228 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827265.195587:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827265.195606:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d08c7840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280612608:192.168.203.124@tcp:400:kworker.0 00000800:00000100:0.0:1787827265.195625:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000800:00000100:0.0:1787827265.195638:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00000400:3.0:1787827265.195652:0:10954:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827217/real 1787827265] req@ffff8ab3d08c6940 x1874671280613120/t0(0) o400->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 224/224 e 0 to 1 dl 1787827233 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827265.195663:0:10954:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8ab3d08c6940 x1874671280613120/t0(0) o400->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 224/224 e 0 to 1 dl 1787827233 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827265.195666:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827265.195692:0:10954:0:(import.c:214:ptlrpc_set_import_discon()) mgc: import ffff8ab3c584f000 already not connected (conn 2, was 1): CONNECTING 00000800:00000100:0.0:1787827265.195707:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827265.195716:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d08c6940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10954:1874671280613120:192.168.203.124@tcp:400:kworker.0 00000800:00000100:0.0:1787827265.195736:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000800:00000100:0.0:1787827265.195770:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827265.195772:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280613632:192.168.203.124@tcp:400:kworker.0 00000800:00000100:0.0:1787827265.195790:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00000400:3.0:1787827265.195818:0:10954:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827221/real 1787827265] req@ffff8ab3f043a940 x1874671280613760/t0(0) o400->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 224/224 e 0 to 1 dl 1787827237 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827265.195820:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827265.195832:0:10954:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8ab3f043a940 x1874671280613760/t0(0) o400->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 224/224 e 0 to 1 dl 1787827237 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827265.195840:0:10954:0:(import.c:214:ptlrpc_set_import_discon()) mgc: import ffff8ab3c584f000 already not connected (conn 2, was 1): CONNECTING 00000800:00000100:0.0:1787827265.195848:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827265.195849:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10954:1874671280613760:192.168.203.124@tcp:400:kworker.0 00000800:00000100:0.0:1787827265.195865:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000800:00000100:0.0:1787827265.195929:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000800:00000100:0.0:1787827265.195945:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000800:00000100:0.0:1787827265.195959:0:10943:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000800:00000100:0.0:1787827265.195973:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 488 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827265.195999:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 488 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827265.196021:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 224 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827265.196043:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 224 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827265.196065:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 224 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827265.196083:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 224 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827265.196099:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 224 0@<0:0>->192.168.203.124@tcp 00000100:00100000:0.0:1787827265.196146:0:10955:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d08c4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:27aedc06-3022-4c11-aafb-6bd27e69b836:10955:1874671280612224:192.168.203.124@tcp:400:kworker.0 00000100:00100000:3.0:1787827265.196170:0:10954:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d08c6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:27aedc06-3022-4c11-aafb-6bd27e69b836:10954:1874671280612096:192.168.203.124@tcp:400:kworker.0 00000100:00080000:0.0:1787827265.196206:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827265.196213:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827265.196217:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827265.196231:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1778 00000100:00080000:0.0:1787827265.196243:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1787827265.196251:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827265.196279:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280614784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827265.196304:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827265.196308:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827265.196309:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827265.196312:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1778 00000100:00080000:0.0:1787827265.196315:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1787827265.196318:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827265.196326:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827265.196336:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280614912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827265.196342:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280614784:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827265.196347:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827265.196350:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827265.196352:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827265.196355:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1778 00000100:00080000:0.0:1787827265.196358:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1787827265.196361:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827265.196364:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827265.196374:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280615040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827265.196380:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827265.196383:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827265.196384:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827265.196386:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1776 00000100:00080000:0.0:1787827265.196389:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 5s 00000800:00000100:3.0:1787827265.196393:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787827265.196395:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827265.196401:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827265.196410:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280615168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827265.196416:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827265.196419:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827265.196420:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827265.196424:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1776 00000100:00080000:0.0:1787827265.196428:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 5s 00000100:00100000:3.0:1787827265.196432:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827265/real 1787827265] req@ffff8ab3f075cf00 x1874671280614784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827265.196434:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827265.196440:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827265.196453:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280615296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827265.196456:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280614784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827265.196474:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827265.196478:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827265.196481:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280614784:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827265.196497:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280614912:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827265.196513:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827265.196521:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827265/real 1787827265] req@ffff8ab3f075f480 x1874671280614912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827265.196535:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280614912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827265.196540:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827265.196542:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827265.196545:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280614912:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827265.196548:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280615040:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827265.196559:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827265.196564:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827265/real 1787827265] req@ffff8ab3f075f0c0 x1874671280615040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827265.196571:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280615040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827265.196576:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827265.196578:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827265.196580:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280615040:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827265.196584:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280615168:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827265.196594:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827265.196600:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827265/real 1787827265] req@ffff8ab3f075e940 x1874671280615168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827265.196607:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280615168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827265.196617:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827265.196619:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827265.196621:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280615168:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827265.196625:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280615296:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827265.196635:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827265.196640:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827265/real 1787827265] req@ffff8ab3f075c000 x1874671280615296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827265.196647:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280615296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827265.196663:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827265.196670:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827265.196682:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280615296:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827265.205811:0:10957:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8ab3d08c4f00 x1874671280612480/t0(0) o400->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 224/224 e 0 to 1 dl 1787827228 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827265.205820:0:10957:0:(import.c:214:ptlrpc_set_import_discon()) mgc: import ffff8ab3c584f000 already not connected (conn 3, was 1): DISCONN 00000100:00100000:2.0:1787827265.205825:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d08c4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10957:1874671280612480:192.168.203.124@tcp:400:kworker.0 00000100:00100000:2.0:1787827265.205832:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d08c43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280611968:192.168.203.124@tcp:400:kworker.0 00000100:00100000:2.0:1787827265.205848:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280613888:192.168.203.124@tcp:400:kworker.0 00000100:00100000:2.0:1787827265.205852:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280614016:192.168.203.124@tcp:400:kworker.0 00000100:00100000:2.0:1787827265.205856:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280613504:192.168.203.124@tcp:400:kworker.0 00000100:00100000:2.0:1787827265.205859:0:10957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d08c5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:27aedc06-3022-4c11-aafb-6bd27e69b836:10957:1874671280612992:192.168.203.124@tcp:400:kworker.0 00010000:00010000:0.0:1787827265.382110:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827267.430106:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827269.478107:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827270.501982:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827270.501993:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827270.501998:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827270.502009:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1819 00000100:00080000:3.0:1787827270.502023:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787827270.502031:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827270.502077:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280615424/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827270.502101:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827270.502105:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827270.502106:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827270.502109:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1819 00000100:00080000:3.0:1787827270.502112:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787827270.502114:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827270.502124:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827270.502134:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280615552/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827270.502142:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827270.502145:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827270.502148:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827270.502155:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1819 00000100:00080000:3.0:1787827270.502158:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787827270.502161:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827270.502165:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827270.502174:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280615680/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827270.502180:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827270.502183:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827270.502184:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827270.502187:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1819 00000100:00080000:3.0:1787827270.502190:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787827270.502192:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827270.502197:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827270.502205:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280615808/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827270.502211:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827270.502214:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827270.502216:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827270.502220:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1819 00000100:00080000:3.0:1787827270.502223:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787827270.502228:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827270.502233:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827270.502246:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280615936/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827271.526054:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827273.574117:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787827273.574247:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.124:988 00000400:00000100:3.0:1787827273.574307:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787827273.574329:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787827273.574384:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787827273.574395:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787827273.574407:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787827273.574419:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00080000:3.0:1787827275.622009:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827275.622021:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827275.622025:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827275.622036:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1825 00000100:00080000:3.0:1787827275.622051:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1787827275.622061:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787827275.622077:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827275.622103:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280616064/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827275.622137:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827275.622140:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827275.622142:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827275.622144:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1825 00000100:00080000:3.0:1787827275.622147:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1787827275.622150:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827275.622159:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827275.622175:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280616192/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827275.622202:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827275.622210:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827275.622213:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827275.622220:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1825 00000100:00080000:3.0:1787827275.622227:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1787827275.622239:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827275.622247:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827275.622265:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280616320/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827275.622275:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827275.622280:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827275.622284:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827275.622296:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1825 00000100:00080000:3.0:1787827275.622306:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1787827275.622317:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827275.622325:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827275.622344:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280616448/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827275.622374:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827275.622379:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827275.622381:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827275.622390:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1825 00000100:00080000:3.0:1787827275.622398:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1787827275.622404:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827275.622413:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827275.622446:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280616576/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000100:2.0:1787827276.646181:0:10942:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.124:988 00000400:00000100:2.0:1787827276.646216:0:10942:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827276.646232:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827276.646283:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827276.646293:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827276.646302:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827276.646312:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00010000:00010000:0.0:1787827277.670070:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827279.718111:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827280.742011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827280.742022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827280.742028:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827280.742041:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1830 00000100:00080000:2.0:1787827280.742056:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827280.742066:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827280.742128:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280616704/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827280.742159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827280.742162:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827280.742163:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827280.742166:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1830 00000100:00080000:2.0:1787827280.742170:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827280.742175:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827280.742185:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827280.742195:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438000 x1874671280616832/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827280.742219:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827280.742225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827280.742228:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827280.742235:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1830 00000100:00080000:2.0:1787827280.742240:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827280.742245:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827280.742252:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827280.742264:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280616960/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827280.742272:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827280.742274:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827280.742276:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827280.742279:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1830 00000100:00080000:2.0:1787827280.742282:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827280.742285:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827280.742291:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827280.742300:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280617088/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827280.742308:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827280.742311:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827280.742313:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827280.742317:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1830 00000100:00080000:2.0:1787827280.742321:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827280.742324:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827280.742327:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827280.742336:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280617216/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827281.766042:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827283.814251:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787827283.814348:0:10944:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.124:988 00000400:00000100:2.0:1787827283.814402:0:10944:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827283.814423:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827283.814465:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827283.814485:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827283.814496:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827283.814511:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00080000:3.0:1787827285.862040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827285.862055:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827285.862061:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827285.862077:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1835 00000100:00080000:3.0:1787827285.862090:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827285.862099:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827285.862141:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280617344/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827285.862143:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827285.862185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827285.862189:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827285.862191:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827285.862196:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1835 00000100:00080000:3.0:1787827285.862201:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827285.862215:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827285.862227:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827285.862240:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280617472/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827285.862259:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827285.862267:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827285.862269:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827285.862275:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1835 00000100:00080000:3.0:1787827285.862278:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827285.862284:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827285.862290:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827285.862306:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280617600/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827285.862322:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827285.862327:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827285.862330:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827285.862336:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1835 00000100:00080000:3.0:1787827285.862343:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827285.862351:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827285.862359:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827285.862379:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280617728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827285.862420:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827285.862426:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827285.862429:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827285.862437:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1835 00000100:00080000:3.0:1787827285.862454:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827285.862462:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827285.862477:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827285.862504:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280617856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827285.862827:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280617600/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827285.862843:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827285.862863:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827285.862868:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280617600:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827285.862876:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280617728:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827285.862949:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827285.862980:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827285/real 1787827285] req@ffff8ab3d0a43480 x1874671280617728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827285.862994:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280617728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827285.863017:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827285.863022:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827285.863025:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280617728:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827285.863033:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280617856:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827285.863073:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827285.863086:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827285/real 1787827285] req@ffff8ab3d0a41e00 x1874671280617856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827285.863097:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280617856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827285.863107:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827285.863111:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827285.863115:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280617856:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827286.886058:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827287.910037:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827288.934087:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827289.958169:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827290.982028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827290.982039:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827290.982044:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827290.982055:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1840 00000100:00080000:2.0:1787827290.982068:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1787827290.982074:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827290.982077:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827290.982122:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280617984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827290.982152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827290.982155:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827290.982156:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827290.982159:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1840 00000100:00080000:2.0:1787827290.982161:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827290.982164:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827290.982175:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827290.982185:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280618112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827290.982194:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827290.982198:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827290.982200:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827290.982203:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280617984:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827290.982205:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1840 00000100:00080000:2.0:1787827290.982208:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827290.982220:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827290.982225:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827290.982237:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280618240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827290.982243:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827290.982246:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827290.982247:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827290.982251:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1840 00000100:00080000:2.0:1787827290.982253:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827290.982257:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827290.982261:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827290.982271:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280618368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827290.982279:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827290.982283:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827290.982285:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827290.982293:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1840 00000800:00000100:1.0:1787827290.982295:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827290.982301:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827290.982308:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827290.982315:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827290.982334:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280618496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827290.982357:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827290/real 1787827290] req@ffff8ab3f0438780 x1874671280617984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827290.982388:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280617984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827290.982399:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827290.982403:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827290.982407:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280617984:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827290.982434:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280618112:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827290.982460:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827290.982474:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827290/real 1787827290] req@ffff8ab3f043ad00 x1874671280618112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827290.982488:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280618112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827290.982511:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827290.982515:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827290.982518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280618112:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827290.982529:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280618240:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827290.982550:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827290.982563:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827290/real 1787827290] req@ffff8ab3f0438f00 x1874671280618240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827290.982576:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280618240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827290.982585:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827290.982589:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827290.982592:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280618240:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827290.982600:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280618368:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827290.982621:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827290.982632:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827290/real 1787827290] req@ffff8ab3f043a580 x1874671280618368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827290.982644:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280618368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827290.982654:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827290.982657:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827290.982660:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280618368:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827290.982667:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280618496:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827290.982687:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827290.982699:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827290/real 1787827290] req@ffff8ab3f043b0c0 x1874671280618496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827290.982710:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280618496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827290.982721:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827290.982724:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827290.982727:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280618496:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827292.006190:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827293.030023:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827294.054018:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827295.078062:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827296.101999:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827296.102010:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827296.102014:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827296.102025:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1845 00000100:00080000:2.0:1787827296.102037:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827296.102046:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787827296.102052:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827296.102092:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280618624/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827296.102123:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827296.102126:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827296.102128:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827296.102131:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1845 00000100:00080000:2.0:1787827296.102133:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827296.102136:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827296.102145:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827296.102156:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280618752/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827296.102164:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827296.102168:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827296.102169:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827296.102175:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1845 00000100:00100000:1.0:1787827296.102176:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280618624:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827296.102177:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827296.102183:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827296.102191:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827296.102203:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438000 x1874671280618880/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827296.102214:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827296.102218:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827296.102221:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827296.102225:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1845 00000100:00080000:2.0:1787827296.102247:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827296.102254:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827296.102260:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827296.102275:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280619008/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827296.102289:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827296.102294:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827296.102296:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827296.102301:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1845 00000100:00080000:2.0:1787827296.102304:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827296.102308:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827296.102311:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280618752:192.168.203.124@tcp:38:kworker.0 00000008:00100000:2.0:1787827296.102322:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827296.102336:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280619136/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827296.102357:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280618880:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827296.102387:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280619008:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827296.102415:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280619136:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827297.126067:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827298.150008:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827299.174036:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827299.174218:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.124:988 00000400:00000100:0.0:1787827299.174282:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827299.174301:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827299.174360:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827299.174371:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827299.174384:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827299.174396:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787827299.174405:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827296/real 1787827299] req@ffff8ab3f0438b40 x1874671280618752/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827299.174433:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280618752/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827299.174463:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827299.174470:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827299.174476:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280618752:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827299.174486:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827296/real 1787827299] req@ffff8ab3f0438000 x1874671280618880/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827299.174501:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438000 x1874671280618880/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827299.174511:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827299.174516:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827299.174520:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280618880:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827299.174528:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827296/real 1787827299] req@ffff8ab3f043bc00 x1874671280619008/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827299.174541:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280619008/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827299.174553:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827299.174556:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827299.174559:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280619008:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827299.174567:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827296/real 1787827299] req@ffff8ab3f04383c0 x1874671280618624/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827299.174595:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280618624/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827299.174605:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827299.174609:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827299.174614:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280618624:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827299.174647:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827296/real 1787827299] req@ffff8ab3f04392c0 x1874671280619136/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827299.174669:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280619136/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827299.174679:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827299.174683:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827299.174685:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280619136:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827300.198078:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827301.222017:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1787827301.222022:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827301.222026:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827301.222031:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827301.222040:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1850 00000100:00080000:3.0:1787827301.222049:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827301.222055:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827301.222084:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280619264/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827301.222104:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827301.222108:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827301.222110:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827301.222112:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1850 00000100:00080000:3.0:1787827301.222115:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827301.222119:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827301.222128:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827301.222136:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280619264:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827301.222143:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280619392/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827301.222155:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827301.222158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827301.222160:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827301.222165:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1850 00000100:00080000:3.0:1787827301.222169:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827301.222174:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827301.222179:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827301.222193:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280619520/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827301.222202:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827301.222218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827301.222222:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827301.222224:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827301.222230:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1850 00000100:00080000:3.0:1787827301.222234:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827301.222240:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827301.222252:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787827301.222253:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827301/real 1787827301] req@ffff8ab3d0a43c00 x1874671280619264/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827301.222273:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280619648/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827301.222277:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280619264/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827301.222298:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827301.222301:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827301.222303:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827301.222308:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827301.222312:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787827301.222317:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1850 00000100:00100000:1.0:1787827301.222322:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280619264:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827301.222324:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827301.222334:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827301.222341:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827301.222354:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280619392:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827301.222356:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280619776/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827301.222387:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827301.222406:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827301/real 1787827301] req@ffff8ab3d0a412c0 x1874671280619392/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827301.222451:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280619392/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827301.222459:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827301.222463:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827301.222479:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280619392:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827301.222486:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280619520:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827301.222507:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827301.222519:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827301/real 1787827301] req@ffff8ab3d0a421c0 x1874671280619520/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827301.222532:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280619520/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827301.222541:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827301.222544:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827301.222548:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280619520:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827301.222555:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280619648:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827301.222574:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827301.222584:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827301/real 1787827301] req@ffff8ab3d0a403c0 x1874671280619648/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827301.222597:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280619648/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827301.222612:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827301.222615:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827301.222618:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280619648:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827301.222636:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280619776:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827301.222654:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827301.222664:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827301/real 1787827301] req@ffff8ab3d0a43c00 x1874671280619776/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827301.222675:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280619776/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827301.222685:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827301.222688:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827301.222691:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280619776:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827302.246180:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827303.270048:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827304.294021:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827305.318038:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827306.342050:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827306.342060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827306.342065:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827306.342076:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1855 00000100:00080000:3.0:1787827306.342088:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827306.342096:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787827306.342125:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827306.342147:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280619904/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827306.342173:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827306.342177:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827306.342178:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827306.342181:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1855 00000100:00080000:3.0:1787827306.342185:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827306.342189:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827306.342201:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827306.342221:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280620032/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827306.342234:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827306.342236:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280619904:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827306.342238:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827306.342244:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827306.342257:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1855 00000100:00080000:3.0:1787827306.342259:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827306.342262:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827306.342267:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827306.342291:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280620160/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827306.342298:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827306.342301:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827306.342303:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827306.342307:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1855 00000100:00080000:3.0:1787827306.342322:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827306.342325:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827306.342330:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787827306.342338:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827306.342347:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280620288/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827306.342361:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827306.342364:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827306.342366:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827306.342370:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1855 00000100:00080000:3.0:1787827306.342373:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827306.342377:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827306.342386:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827306/real 1787827306] req@ffff8ab3d0a41e00 x1874671280619904/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827306.342408:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827306.342423:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280620416/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827306.342438:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280619904/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827306.342448:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827306.342455:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827306.342459:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280619904:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827306.342489:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280620032:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827306.342507:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827306.342516:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827306/real 1787827306] req@ffff8ab3d0a43480 x1874671280620032/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827306.342525:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280620032/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827306.342530:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827306.342533:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827306.342535:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280620032:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827306.342539:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280620160:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827306.342549:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827306.342555:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827306/real 1787827306] req@ffff8ab3d0a43840 x1874671280620160/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827306.342562:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280620160/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827306.342567:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827306.342569:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827306.342570:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280620160:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827306.342574:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280620288:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827306.342585:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827306.342591:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827306/real 1787827306] req@ffff8ab3d0a430c0 x1874671280620288/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827306.342598:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280620288/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827306.342604:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827306.342606:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827306.342608:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280620288:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827306.342612:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280620416:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827306.342621:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827306.342626:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827306/real 1787827306] req@ffff8ab3d0a40000 x1874671280620416/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827306.342633:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280620416/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827306.342639:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827306.342641:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827306.342643:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280620416:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827307.366073:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827308.390043:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827309.414028:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827310.438101:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827311.462008:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827311.462025:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827311.462031:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827311.462045:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1861 00000100:00080000:3.0:1787827311.462058:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827311.462071:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787827311.462090:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827311.462129:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280620544/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827311.462155:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827311.462158:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827311.462159:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827311.462163:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1861 00000100:00080000:3.0:1787827311.462166:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827311.462171:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827311.462185:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827311.462201:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280620544:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827311.462202:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280620672/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827311.462221:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827311.462224:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827311.462227:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827311.462232:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1861 00000100:00080000:3.0:1787827311.462236:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827311.462241:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787827311.462284:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787827311.462292:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827311.462308:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280620800/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827311.462321:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827311.462325:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827311.462328:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827311.462333:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1861 00000100:00080000:3.0:1787827311.462339:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827311.462340:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827311/real 1787827311] req@ffff8ab3d0a40000 x1874671280620544/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827311.462349:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827311.462370:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787827311.462391:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280620544/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827311.462391:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280620928/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827311.462425:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827311.462426:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827311.462431:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787827311.462434:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827311.462437:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827311.462445:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280620544:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827311.462447:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1861 00000100:00080000:3.0:1787827311.462454:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827311.462459:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827311.462465:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827311.462474:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280620672:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827311.462481:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280621056/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827311.462503:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827311.462516:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827311/real 1787827311] req@ffff8ab3d0a430c0 x1874671280620672/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827311.462525:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280620672/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827311.462531:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827311.462534:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827311.462536:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280620672:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827311.462539:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280620800:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827311.462550:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827311.462556:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827311/real 1787827311] req@ffff8ab3d0a43840 x1874671280620800/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827311.462563:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280620800/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827311.462568:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827311.462570:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827311.462573:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280620800:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827311.462577:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280620928:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827311.462588:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827311.462593:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827311/real 1787827311] req@ffff8ab3d0a43480 x1874671280620928/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827311.462600:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280620928/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827311.462608:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827311.462610:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827311.462612:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280620928:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827311.462621:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280621056:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827311.462631:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827311.462637:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827311/real 1787827311] req@ffff8ab3d0a41e00 x1874671280621056/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827311.462645:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280621056/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827311.462652:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827311.462654:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827311.462655:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280621056:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827312.486069:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827313.510078:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827314.534056:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827315.558024:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827316.582006:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827316.582017:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827316.582022:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827316.582040:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1866 00000100:00080000:3.0:1787827316.582052:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827316.582060:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787827316.582099:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827316.582105:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280621184/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827316.582131:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827316.582133:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827316.582135:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827316.582138:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1866 00000100:00080000:3.0:1787827316.582141:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827316.582144:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827316.582153:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827316.582162:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280621312/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827316.582177:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280621184:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827316.582177:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827316.582182:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827316.582184:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827316.582191:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1866 00000100:00080000:3.0:1787827316.582193:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827316.582195:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827316.582198:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827316.582207:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280621440/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827316.582213:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827316.582216:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827316.582218:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827316.582220:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1866 00000100:00080000:3.0:1787827316.582223:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827316.582226:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827316.582229:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827316.582244:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280621568/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827316.582250:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827316.582253:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827316.582255:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827316.582257:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1866 00000100:00080000:3.0:1787827316.582260:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827316.582264:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827316.582267:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827316.582275:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280621696/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827316.582343:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280621312:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827316.582382:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280621440:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827316.582410:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280621568:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827316.582438:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280621696:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827317.606030:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827318.630058:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827319.654061:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787827319.654175:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787827319.654217:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827319.654238:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827319.654280:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827319.654293:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827319.654320:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827319.654330:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787827319.654341:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827316/real 1787827319] req@ffff8ab3d0a403c0 x1874671280621312/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827319.654374:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280621312/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827319.654390:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827319.654396:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827319.654401:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280621312:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827319.654411:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827316/real 1787827319] req@ffff8ab3d0a421c0 x1874671280621440/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827319.654425:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280621440/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827319.654434:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827319.654439:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827319.654443:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280621440:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827319.654450:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827316/real 1787827319] req@ffff8ab3d0a412c0 x1874671280621568/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827319.654465:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280621568/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827319.654478:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827319.654482:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827319.654486:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280621568:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827319.654493:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827316/real 1787827319] req@ffff8ab3d0a41e00 x1874671280621696/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827319.654506:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280621696/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827319.654516:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827319.654536:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827319.654540:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280621696:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827319.654548:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827316/real 1787827319] req@ffff8ab3d0a43c00 x1874671280621184/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827319.654562:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280621184/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827319.654571:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827319.654575:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827319.654578:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280621184:192.168.203.124@tcp:250:kworker.0 00010000:00010000:0.0:1787827320.678057:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827321.701983:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827321.702007:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827321.702013:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827321.702026:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1871 00000100:00080000:2.0:1787827321.702036:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827321.702045:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827321.702088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280621824/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827321.702111:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827321.702114:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827321.702122:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827321.702125:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827321.702133:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1871 00000100:00080000:2.0:1787827321.702139:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827321.702145:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280621824:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827321.702147:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827321.702164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827321.702181:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280621952/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827321.702200:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827321.702204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827321.702206:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827321.702211:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1871 00000100:00080000:2.0:1787827321.702224:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827321.702230:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827321.702236:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827321.702237:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827321.702252:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280622080/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827321.702264:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827321.702269:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827321.702271:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827321.702276:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1871 00000100:00080000:2.0:1787827321.702295:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827321.702301:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827321.702308:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827321.702322:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280622208/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827321.702332:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827321/real 1787827321] req@ffff8ab3f0439a40 x1874671280621824/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827321.702335:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827321.702343:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827321.702348:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827321.702358:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1871 00000100:00080000:2.0:1787827321.702363:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827321.702370:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827321.702371:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280621824/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827321.702377:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787827321.702391:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827321.702396:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280622336/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827321.702401:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827321.702411:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280621824:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827321.702449:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280621952:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827321.702477:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827321.702492:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827321/real 1787827321] req@ffff8ab3f0439680 x1874671280621952/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827321.702512:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280621952/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827321.702521:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827321.702525:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827321.702528:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280621952:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827321.702535:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280622080:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827321.702557:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827321.702569:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827321/real 1787827321] req@ffff8ab3f043a1c0 x1874671280622080/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827321.702582:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280622080/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827321.702596:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827321.702600:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827321.702603:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280622080:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827321.702609:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280622208:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827321.702628:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827321.702639:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827321/real 1787827321] req@ffff8ab3f043a940 x1874671280622208/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827321.702664:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280622208/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827321.702676:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827321.702685:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827321.702689:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280622208:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827321.702696:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280622336:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827321.702718:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827321.702731:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827321/real 1787827321] req@ffff8ab3f043b480 x1874671280622336/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827321.702742:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280622336/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827321.702753:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827321.702756:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827321.702760:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280622336:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827322.726151:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827323.750043:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827324.774054:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827325.798057:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827326.821959:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827326.821967:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827326.821970:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827326.821979:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1876 00000100:00080000:2.0:1787827326.821988:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827326.821994:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827326.822027:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280622464/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827326.822047:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827326.822050:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827326.822052:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827326.822054:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1876 00000100:00080000:2.0:1787827326.822056:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827326.822059:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827326.822067:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827326.822075:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280622592/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827326.822086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827326.822088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827326.822090:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827326.822092:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1876 00000100:00080000:2.0:1787827326.822093:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827326.822096:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827326.822098:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827326.822104:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280622720/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827326.822110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827326.822111:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827326.822112:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827326.822115:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1876 00000100:00080000:2.0:1787827326.822117:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827326.822119:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827326.822122:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827326.822128:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280622848/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827326.822133:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827326.822135:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827326.822137:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827326.822139:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1876 00000100:00080000:2.0:1787827326.822141:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827326.822144:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827326.822146:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827326.822152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280622976/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827326.822303:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280622464:192.168.203.124@tcp:250:kworker.0 00010000:00010000:0.0:1787827326.822337:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:1.0:1787827326.822360:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827326.822416:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827326/real 1787827326] req@ffff8ab3f0439e00 x1874671280622464/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827326.822430:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280622464/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827326.822440:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827326.822445:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827326.822449:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280622464:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827326.822457:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280622592:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827326.822481:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827326.822497:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827326/real 1787827326] req@ffff8ab3f043b840 x1874671280622592/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827326.822510:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280622592/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827326.822518:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827326.822521:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827326.822525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280622592:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827326.822532:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280622720:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827326.822547:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827326.822556:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827326/real 1787827326] req@ffff8ab3f043b480 x1874671280622720/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827326.822564:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280622720/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827326.822570:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827326.822573:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827326.822577:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280622720:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827326.822583:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280622848:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827326.822604:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827326.822616:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827326/real 1787827326] req@ffff8ab3f043a940 x1874671280622848/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827326.822629:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280622848/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827326.822639:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827326.822643:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827326.822646:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280622848:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827326.822653:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280622976:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827326.822670:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827326.822679:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827326/real 1787827326] req@ffff8ab3f043a1c0 x1874671280622976/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827326.822686:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280622976/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827326.822693:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827326.822695:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827326.822697:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280622976:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827327.846125:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827328.870184:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827329.894165:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827330.918147:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827331.942036:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827331.942047:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827331.942051:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827331.942061:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1881 00000100:00080000:3.0:1787827331.942072:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827331.942081:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827331.942120:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280623104/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827331.942154:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827331.942157:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827331.942159:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827331.942163:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1881 00000100:00080000:3.0:1787827331.942166:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827331.942169:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827331.942179:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827331.942190:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280623232/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827331.942198:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827331.942204:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827331.942206:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827331.942224:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1881 00000100:00080000:3.0:1787827331.942229:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827331.942240:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280623104:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827331.942240:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787827331.942242:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:3.0:1787827331.942253:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827331.942275:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280623360/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827331.942295:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827331.942300:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827331.942302:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827331.942309:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1881 00000100:00080000:3.0:1787827331.942316:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827331.942322:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827331.942324:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827331.942333:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827331.942348:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280623488/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827331.942356:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827331.942359:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827331.942361:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827331.942365:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1881 00000100:00080000:3.0:1787827331.942370:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827331.942374:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827331.942380:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827331.942387:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827331/real 1787827331] req@ffff8ab3d0a43480 x1874671280623104/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827331.942391:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280623616/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827331.942462:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280623104/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827331.942478:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827331.942483:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827331.942488:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280623104:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827331.942497:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280623232:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827331.942523:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827331.942537:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827331/real 1787827331] req@ffff8ab3d0a43840 x1874671280623232/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827331.942551:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280623232/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827331.942560:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827331.942564:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827331.942567:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280623232:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827331.942593:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280623360:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827331.942615:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827331.942628:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827331/real 1787827331] req@ffff8ab3d0a430c0 x1874671280623360/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827331.942641:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280623360/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827331.942649:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827331.942653:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827331.942656:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280623360:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827331.942663:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280623488:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827331.942683:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827331.942694:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827331/real 1787827331] req@ffff8ab3d0a40000 x1874671280623488/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827331.942707:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280623488/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827331.942718:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827331.942722:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827331.942725:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280623488:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827331.942732:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280623616:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827331.942749:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827331.942759:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827331/real 1787827331] req@ffff8ab3d0a43c00 x1874671280623616/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827331.942771:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280623616/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827331.942782:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827331.942785:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827331.942788:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280623616:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827332.966127:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827333.990089:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827335.014088:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827336.038616:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827337.062023:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827337.062035:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827337.062040:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827337.062052:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1886 00000100:00080000:3.0:1787827337.062064:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827337.062073:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787827337.062107:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827337.062112:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280623744/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827337.062142:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827337.062147:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827337.062149:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827337.062154:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1886 00000100:00080000:3.0:1787827337.062158:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827337.062164:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827337.062188:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280623744:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787827337.062188:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827337.062207:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280623872/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827337.062221:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827337.062224:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827337.062226:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827337.062230:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1886 00000100:00080000:3.0:1787827337.062237:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827337.062242:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787827337.062247:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787827337.062251:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827337.062267:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280624000/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827337.062280:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827337.062283:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827337.062285:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827337.062292:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827337/real 1787827337] req@ffff8ab3d0a41e00 x1874671280623744/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827337.062292:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1886 00000100:00080000:3.0:1787827337.062300:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827337.062308:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827337.062314:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280623744/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827337.062315:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:2.0:1787827337.062329:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827337.062332:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827337.062337:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280624128/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827337.062338:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280623744:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827337.062353:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827337.062358:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827337.062361:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827337.062365:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280623872:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787827337.062370:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1886 00000100:00080000:3.0:1787827337.062377:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827337.062382:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787827337.062388:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787827337.062388:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827337.062400:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827337/real 1787827337] req@ffff8ab3d0a412c0 x1874671280623872/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827337.062408:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280624256/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827337.062414:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280623872/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827337.062457:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827337.062461:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827337.062465:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280623872:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827337.062470:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280624000:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827337.062484:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827337.062491:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827337/real 1787827337] req@ffff8ab3d0a421c0 x1874671280624000/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827337.062502:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280624000/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827337.062507:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827337.062509:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827337.062511:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280624000:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827337.062515:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280624128:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827337.062524:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827337.062530:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827337/real 1787827337] req@ffff8ab3d0a403c0 x1874671280624128/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827337.062536:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280624128/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827337.062542:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827337.062544:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827337.062547:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280624128:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827337.062557:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280624256:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827337.062566:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827337.062572:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827337/real 1787827337] req@ffff8ab3d0a41e00 x1874671280624256/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827337.062579:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280624256/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827337.062585:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827337.062587:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827337.062588:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280624256:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827338.086064:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827339.110000:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827340.134085:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827341.158074:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827342.181957:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827342.181966:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827342.181970:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827342.181978:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1891 00000100:00080000:3.0:1787827342.181991:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827342.181997:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787827342.182039:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827342.182050:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280624384/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827342.182072:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827342.182075:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827342.182076:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827342.182079:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1891 00000100:00080000:3.0:1787827342.182082:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827342.182085:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827342.182095:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827342.182104:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280624512/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827342.182109:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280624384:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827342.182118:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827342.182122:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827342.182123:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827342.182127:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1891 00000100:00080000:3.0:1787827342.182129:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827342.182132:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827342.182136:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827342.182161:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280624640/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827342.182169:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827342.182171:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827342.182173:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827342.182176:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1891 00000100:00080000:3.0:1787827342.182179:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827342.182182:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827342.182186:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787827342.182189:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827342.182198:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280624768/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827342.182206:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827342.182209:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827342.182210:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827342.182213:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1891 00000100:00080000:3.0:1787827342.182216:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827342.182218:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827342.182222:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827342.182231:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280624896/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827342.182236:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827342/real 1787827342] req@ffff8ab3d0a43c00 x1874671280624384/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827342.182256:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280624384/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827342.182281:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827342.182284:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827342.182287:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280624384:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827342.182306:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280624512:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827342.182322:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827342.182348:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827342/real 1787827342] req@ffff8ab3d0a40000 x1874671280624512/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827342.182357:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280624512/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827342.182363:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827342.182366:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827342.182368:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280624512:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827342.182373:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280624640:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827342.182384:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827342.182390:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827342/real 1787827342] req@ffff8ab3d0a430c0 x1874671280624640/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827342.182416:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280624640/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827342.182425:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827342.182429:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827342.182433:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280624640:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827342.182440:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280624768:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827342.182464:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827342.182479:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827342/real 1787827342] req@ffff8ab3d0a43840 x1874671280624768/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827342.182494:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280624768/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827342.182506:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827342.182510:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827342.182514:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280624768:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827342.182522:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280624896:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827342.182554:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827342.182564:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827342/real 1787827342] req@ffff8ab3d0a43480 x1874671280624896/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827342.182573:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280624896/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827342.182580:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827342.182583:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827342.182585:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280624896:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827343.206188:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827344.230011:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827345.254075:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827346.278039:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827347.302011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827347.302023:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827347.302027:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827347.302038:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1896 00000100:00080000:2.0:1787827347.302048:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1787827347.302055:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827347.302056:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827347.302098:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280625024/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827347.302140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827347.302145:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827347.302147:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827347.302152:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1896 00000100:00080000:2.0:1787827347.302157:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827347.302163:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827347.302173:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280625024:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787827347.302174:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827347.302194:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280625152/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827347.302206:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827347.302210:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827347.302212:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827347.302217:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1896 00000100:00080000:2.0:1787827347.302223:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827347.302227:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787827347.302231:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787827347.302236:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827347.302253:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280625280/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827347.302262:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827347.302266:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827347.302269:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827347.302274:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827347/real 1787827347] req@ffff8ab3f0439680 x1874671280625024/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827347.302277:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1896 00000100:00080000:2.0:1787827347.302285:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827347.302292:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827347.302296:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280625024/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827347.302301:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787827347.302315:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827347.302317:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280625408/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827347.302321:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827347.302329:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280625024:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827347.302342:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827347.302346:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827347.302348:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827347.302354:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1896 00000100:00100000:1.0:1787827347.302359:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280625152:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787827347.302360:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827347.302370:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827347.302377:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827347.302390:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280625536/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827347.302395:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827347.302418:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827347/real 1787827347] req@ffff8ab3f0439a40 x1874671280625152/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827347.302429:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280625152/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827347.302436:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827347.302438:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827347.302440:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280625152:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827347.302445:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280625280:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827347.302461:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827347.302467:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827347/real 1787827347] req@ffff8ab3f04392c0 x1874671280625280/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827347.302475:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280625280/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827347.302480:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827347.302482:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827347.302483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280625280:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827347.302487:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280625408:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827347.302497:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827347.302503:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827347/real 1787827347] req@ffff8ab3f04383c0 x1874671280625408/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827347.302510:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280625408/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827347.302516:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827347.302518:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827347.302519:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280625408:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827347.302530:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280625536:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827347.302539:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827347.302545:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827347/real 1787827347] req@ffff8ab3f043bc00 x1874671280625536/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827347.302553:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280625536/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827347.302559:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827347.302560:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827347.302562:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280625536:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827348.326013:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827349.350088:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827350.374004:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827351.398085:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827352.422060:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1787827352.422062:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827352.422085:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827352.422090:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827352.422115:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1902 00000100:00080000:0.0:1787827352.422124:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827352.422130:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827352.422162:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280625664/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827352.422184:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827352.422186:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827352.422188:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827352.422190:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1902 00000100:00080000:0.0:1787827352.422193:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827352.422195:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827352.422204:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827352.422213:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280625792/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827352.422218:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280625664:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827352.422227:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827352.422230:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827352.422231:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827352.422235:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1902 00000100:00080000:0.0:1787827352.422238:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827352.422240:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827352.422243:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827352.422253:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280625920/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827352.422259:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827352.422262:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827352.422264:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827352.422267:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1902 00000100:00080000:0.0:1787827352.422269:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827352.422272:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827352.422275:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827352.422283:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280626048/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827352.422290:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827352.422292:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827352.422294:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827352.422297:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1902 00000100:00080000:0.0:1787827352.422300:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827352.422304:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827352.422307:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827352.422315:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280626176/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827352.422319:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280625792:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827352.422348:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280625920:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827352.422382:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280626048:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827352.422401:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280626176:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827353.446053:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827354.470033:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827355.494235:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787827355.494392:0:10942:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:1.0:1787827355.494461:0:10942:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827355.494483:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787827355.494538:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787827355.494550:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787827355.494573:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:2.0:1787827355.494593:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827352/real 1787827355] req@ffff8ab3f075de00 x1874671280625792/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787827355.494595:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:2.0:1787827355.494637:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280625792/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827355.494651:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827355.494657:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827355.494662:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280625792:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827355.494678:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827352/real 1787827355] req@ffff8ab3f075c000 x1874671280625920/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827355.494713:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280625920/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827355.494723:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827355.494727:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827355.494730:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280625920:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827355.494738:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827352/real 1787827355] req@ffff8ab3f075e940 x1874671280626048/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827355.494753:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280626048/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827355.494777:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827355.494781:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827355.494785:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280626048:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827355.494793:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827352/real 1787827355] req@ffff8ab3f075f840 x1874671280625664/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827355.494807:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280625664/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827355.494815:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827355.494818:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827355.494821:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280625664:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827355.494852:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827352/real 1787827355] req@ffff8ab3f075f0c0 x1874671280626176/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827355.494867:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280626176/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827355.494878:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827355.494930:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827355.494935:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280626176:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827356.518081:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827357.542083:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827357.542086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827357.542096:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827357.542100:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827357.542109:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1907 00000100:00080000:2.0:1787827357.542119:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827357.542125:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827357.542156:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438000 x1874671280626304/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827357.542182:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827357.542187:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827357.542189:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827357.542194:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1907 00000100:00080000:2.0:1787827357.542199:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827357.542204:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827357.542231:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827357.542240:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280626304:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827357.542244:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280626432/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827357.542255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827357.542258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827357.542260:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827357.542264:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1907 00000100:00080000:2.0:1787827357.542267:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827357.542271:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827357.542276:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827357.542286:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280626560/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827357.542294:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827357.542297:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827357.542299:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827357.542300:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827357.542303:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1907 00000100:00080000:2.0:1787827357.542306:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827357.542310:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827357.542314:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827357.542322:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280626688/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827357.542331:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827357.542334:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827357.542336:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827357.542340:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1907 00000100:00100000:1.0:1787827357.542343:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827357/real 1787827357] req@ffff8ab3f0438000 x1874671280626304/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827357.542369:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827357.542374:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827357.542378:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438000 x1874671280626304/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827357.542380:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787827357.542391:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827357.542391:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280626816/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827357.542397:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827357.542402:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280626304:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827357.542440:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280626432:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827357.542463:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827357.542474:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827357/real 1787827357] req@ffff8ab3f0438b40 x1874671280626432/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827357.542485:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280626432/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827357.542494:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827357.542497:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827357.542501:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280626432:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827357.542508:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280626560:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827357.542529:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827357.542541:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827357/real 1787827357] req@ffff8ab3f043b0c0 x1874671280626560/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827357.542554:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280626560/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827357.542569:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827357.542572:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827357.542575:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280626560:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827357.542582:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280626688:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827357.542603:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827357.542615:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827357/real 1787827357] req@ffff8ab3f043a580 x1874671280626688/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827357.542627:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280626688/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827357.542638:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827357.542641:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827357.542644:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280626688:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827357.542650:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280626816:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827357.542670:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827357.542681:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827357/real 1787827357] req@ffff8ab3f0438f00 x1874671280626816/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827357.542693:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280626816/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827357.542703:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827357.542706:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827357.542710:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280626816:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827358.566046:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827359.590098:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827360.614032:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827361.638041:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827362.662013:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827362.662022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827362.662026:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827362.662034:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1912 00000100:00080000:3.0:1787827362.662043:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827362.662050:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827362.662061:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827362.662086:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280626944/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827362.662107:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827362.662109:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827362.662111:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827362.662113:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1912 00000100:00080000:3.0:1787827362.662116:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827362.662119:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827362.662127:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827362.662139:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280627072/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827362.662140:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280626944:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827362.662155:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827362.662157:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827362.662159:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827362.662161:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1912 00000100:00080000:3.0:1787827362.662163:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827362.662166:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827362.662169:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827362.662178:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280627200/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827362.662189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827362.662191:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827362.662192:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827362.662195:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1912 00000100:00080000:3.0:1787827362.662197:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827362.662201:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827362.662202:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827362.662208:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827362.662217:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280627328/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827362.662223:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827362.662225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827362.662227:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827362.662229:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1912 00000100:00080000:3.0:1787827362.662232:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827362.662235:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827362.662238:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827362.662246:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827362/real 1787827362] req@ffff8ab3d0a43480 x1874671280626944/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827362.662248:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280627456/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827362.662281:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280626944/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827362.662291:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827362.662294:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827362.662297:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280626944:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827362.662315:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280627072:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827362.662328:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827362.662336:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827362/real 1787827362] req@ffff8ab3d0a43840 x1874671280627072/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827362.662344:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280627072/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827362.662350:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827362.662352:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827362.662354:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280627072:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827362.662358:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280627200:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827362.662367:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827362.662373:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827362/real 1787827362] req@ffff8ab3d0a430c0 x1874671280627200/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827362.662380:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280627200/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827362.662385:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827362.662387:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827362.662389:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280627200:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827362.662392:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280627328:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827362.662402:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827362.662407:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827362/real 1787827362] req@ffff8ab3d0a40000 x1874671280627328/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827362.662415:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280627328/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827362.662421:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827362.662423:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827362.662425:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280627328:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827362.662428:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280627456:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827362.662437:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827362.662442:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827362/real 1787827362] req@ffff8ab3d0a43c00 x1874671280627456/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827362.662449:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280627456/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827362.662456:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827362.662458:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827362.662459:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280627456:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827363.686024:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827364.710083:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827365.734035:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827366.758159:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827367.781985:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827367.781996:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827367.782000:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827367.782011:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1917 00000100:00080000:3.0:1787827367.782025:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827367.782035:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787827367.782059:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827367.782083:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280627584/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827367.782105:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827367.782108:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827367.782109:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827367.782112:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1917 00000100:00080000:3.0:1787827367.782115:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827367.782118:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827367.782128:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827367.782137:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280627712/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827367.782142:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280627584:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827367.782154:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827367.782157:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827367.782159:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827367.782163:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1917 00000100:00080000:3.0:1787827367.782166:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827367.782168:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827367.782172:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827367.782182:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280627840/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827367.782190:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827367.782193:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827367.782194:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827367.782197:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1917 00000100:00080000:3.0:1787827367.782204:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827367.782206:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827367.782210:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787827367.782215:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827367.782221:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280627968/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827367.782229:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827367.782232:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827367.782234:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827367.782237:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1917 00000100:00080000:3.0:1787827367.782239:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827367.782242:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827367.782245:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827367.782258:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280628096/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827367.782268:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827367/real 1787827367] req@ffff8ab3d0a41e00 x1874671280627584/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827367.782284:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280627584/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827367.782293:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827367.782296:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827367.782299:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280627584:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827367.782321:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280627712:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827367.782335:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827367.782343:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827367/real 1787827367] req@ffff8ab3d0a403c0 x1874671280627712/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827367.782351:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280627712/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827367.782357:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827367.782358:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827367.782360:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280627712:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827367.782365:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280627840:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827367.782375:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827367.782381:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827367/real 1787827367] req@ffff8ab3d0a421c0 x1874671280627840/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827367.782388:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280627840/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827367.782393:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827367.782395:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827367.782396:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280627840:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827367.782400:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280627968:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827367.782410:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827367.782415:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827367/real 1787827367] req@ffff8ab3d0a412c0 x1874671280627968/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827367.782422:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280627968/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827367.782428:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827367.782430:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827367.782432:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280627968:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827367.782435:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280628096:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827367.782444:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827367.782449:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827367/real 1787827367] req@ffff8ab3d0a43c00 x1874671280628096/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827367.782457:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280628096/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827367.782464:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827367.782465:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827367.782467:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280628096:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827368.806106:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827369.830107:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827370.854114:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827371.878115:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827372.902088:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827372.902106:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827372.902131:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827372.902149:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1922 00000100:00080000:0.0:1787827372.902167:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787827372.902174:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827372.902180:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827372.902238:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280628224/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827372.902273:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827372.902285:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827372.902305:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827372.902312:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1922 00000100:00080000:0.0:1787827372.902319:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827372.902325:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827372.902340:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827372.902357:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280628224:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827372.902399:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280628352/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827372.902427:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827372.902450:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827372.902454:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827372.902462:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1922 00000100:00080000:0.0:1787827372.902483:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827372.902493:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787827372.902495:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827372.902509:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827372.902537:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280628480/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827372.902556:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827372.902564:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827372/real 1787827372] req@ffff8ab3f075f480 x1874671280628224/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.902566:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827372.902575:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827372.902590:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1922 00000100:00080000:0.0:1787827372.902614:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827372.902618:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280628224/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.902627:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827372.902646:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787827372.902652:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827372.902662:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827372.902675:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280628608/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827372.902676:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280628224:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827372.902703:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827372.902708:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827372.902710:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827372.902717:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1922 00000100:00080000:0.0:1787827372.902722:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827372.902726:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280628352:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787827372.902733:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827372.902743:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827372.902761:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280628736/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827372.902776:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827372.902817:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827372/real 1787827372] req@ffff8ab3f075cf00 x1874671280628352/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827372.902832:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280628352/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827372.902848:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827372.902852:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827372.902856:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280628352:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827372.902862:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280628480:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827372.902879:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827372.902888:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827372/real 1787827372] req@ffff8ab3f075e580 x1874671280628480/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827372.902899:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280628480/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827372.902907:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827372.902910:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827372.902913:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280628480:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827372.902918:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280628608:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827372.902983:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827372.902994:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827372/real 1787827372] req@ffff8ab3f075cb40 x1874671280628608/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827372.903006:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280628608/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827372.903016:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827372.903019:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827372.903023:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280628608:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827372.903040:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280628736:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827372.903056:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827372.903066:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827372/real 1787827372] req@ffff8ab3f075da40 x1874671280628736/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827372.903077:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280628736/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827372.903087:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827372.903090:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827372.903092:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280628736:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827373.926027:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827374.950096:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827375.974025:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827376.998047:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827378.022064:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827378.022077:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827378.022083:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827378.022098:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1927 00010000:00010000:2.0:1787827378.022111:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827378.022113:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827378.022128:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827378.022183:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280628864/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827378.022232:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827378.022240:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827378.022244:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827378.022254:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1927 00000100:00080000:3.0:1787827378.022264:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827378.022271:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827378.022285:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827378.022289:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280628864:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827378.022317:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280628992/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827378.022343:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827378.022350:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827378.022355:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827378.022363:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1927 00000100:00080000:3.0:1787827378.022386:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827378.022394:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827378.022403:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827378.022414:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827378.022444:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280629120/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827378.022465:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827378/real 1787827378] req@ffff8ab3d0a40000 x1874671280628864/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827378.022469:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827378.022486:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827378.022494:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827378.022509:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1927 00000100:00100000:1.0:1787827378.022512:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280628864/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827378.022518:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827378.022528:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827378.022542:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787827378.022550:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827378.022559:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827378.022570:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280629248/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827378.022571:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280628864:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827378.022596:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827378.022602:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827378.022605:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827378.022613:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1927 00000100:00080000:3.0:1787827378.022620:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827378.022621:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280628992:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787827378.022635:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827378.022644:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827378.022661:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280629376/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827378.022688:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827378.022715:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827378/real 1787827378] req@ffff8ab3d0a430c0 x1874671280628992/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827378.022740:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280628992/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827378.022751:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827378.022755:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827378.022760:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280628992:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827378.022767:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280629120:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827378.022789:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827378.022801:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827378/real 1787827378] req@ffff8ab3d0a43840 x1874671280629120/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827378.022815:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280629120/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827378.022825:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827378.022829:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827378.022832:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280629120:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827378.022839:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280629248:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827378.022857:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827378.022867:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827378/real 1787827378] req@ffff8ab3d0a43480 x1874671280629248/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827378.022881:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280629248/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827378.022949:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827378.022953:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827378.022957:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280629248:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827378.022978:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280629376:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827378.023002:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827378.023016:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827378/real 1787827378] req@ffff8ab3d0a40000 x1874671280629376/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827378.023031:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280629376/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827378.023042:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827378.023046:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827378.023050:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280629376:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827379.046059:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827380.070076:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827381.094074:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827382.118069:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827383.141978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827383.141988:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827383.141991:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827383.142000:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1932 00000100:00080000:3.0:1787827383.142008:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787827383.142015:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827383.142015:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827383.142050:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280629504/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827383.142070:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827383.142075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827383.142076:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827383.142078:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1932 00000100:00080000:3.0:1787827383.142081:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827383.142084:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827383.142092:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827383.142101:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280629632/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827383.142109:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827383.142112:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280629504:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827383.142112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827383.142116:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827383.142123:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1932 00000100:00080000:3.0:1787827383.142126:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827383.142128:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827383.142131:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827383.142140:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280629760/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827383.142156:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827383.142159:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827383.142160:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827383.142163:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1932 00000100:00080000:3.0:1787827383.142177:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827383.142180:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827383.142184:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787827383.142187:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827383.142196:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280629888/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827383.142205:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827383.142207:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827383.142208:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827383.142211:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1932 00000100:00080000:3.0:1787827383.142222:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827383.142225:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827383.142228:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827383.142236:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280630016/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827383.142243:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827383/real 1787827383] req@ffff8ab3d0a43c00 x1874671280629504/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827383.142260:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280629504/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827383.142268:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827383.142271:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827383.142274:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280629504:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827383.142304:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280629632:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827383.142318:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827383.142327:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827383/real 1787827383] req@ffff8ab3d0a412c0 x1874671280629632/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827383.142335:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280629632/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827383.142340:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827383.142342:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827383.142345:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280629632:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827383.142348:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280629760:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827383.142358:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827383.142363:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827383/real 1787827383] req@ffff8ab3d0a421c0 x1874671280629760/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827383.142370:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280629760/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827383.142375:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827383.142377:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827383.142378:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280629760:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827383.142382:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280629888:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827383.142392:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827383.142398:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827383/real 1787827383] req@ffff8ab3d0a403c0 x1874671280629888/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827383.142405:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280629888/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827383.142422:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827383.142424:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827383.142426:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280629888:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827383.142429:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280630016:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827383.142438:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827383.142444:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827383/real 1787827383] req@ffff8ab3d0a41e00 x1874671280630016/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827383.142451:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280630016/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827383.142457:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827383.142459:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827383.142460:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280630016:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827384.166078:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827385.190179:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827386.214039:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827387.238088:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827388.262040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827388.262053:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827388.262059:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827388.262071:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1937 00000100:00080000:3.0:1787827388.262101:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827388.262110:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:1.0:1787827388.262112:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827388.262158:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280630144/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827388.262187:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827388.262191:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827388.262193:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827388.262197:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1937 00000100:00080000:3.0:1787827388.262202:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827388.262206:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827388.262219:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827388.262233:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280630272/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827388.262242:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280630144:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827388.262253:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827388.262258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827388.262260:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827388.262265:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1937 00000100:00080000:3.0:1787827388.262268:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827388.262272:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827388.262277:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827388.262292:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280630400/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827388.262301:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827388.262305:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827388.262307:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827388.262311:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1937 00000100:00080000:3.0:1787827388.262316:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827388.262319:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787827388.262328:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787827388.262331:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827388.262348:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280630528/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827388.262366:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827388.262370:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827388.262373:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827388.262380:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1937 00000100:00100000:1.0:1787827388.262387:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827388/real 1787827388] req@ffff8ab3d0a41e00 x1874671280630144/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827388.262390:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827388.262404:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827388.262421:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827388.262432:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280630144/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827388.262446:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280630656/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827388.262449:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827388.262461:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827388.262472:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280630144:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827388.262507:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280630272:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827388.262530:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827388.262544:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827388/real 1787827388] req@ffff8ab3d0a403c0 x1874671280630272/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827388.262565:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280630272/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827388.262578:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827388.262583:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827388.262589:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280630272:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827388.262600:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280630400:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827388.262632:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827388.262649:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827388/real 1787827388] req@ffff8ab3d0a421c0 x1874671280630400/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827388.262661:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280630400/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827388.262676:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827388.262680:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827388.262683:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280630400:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827388.262689:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280630528:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827388.262709:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827388.262718:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827388/real 1787827388] req@ffff8ab3d0a412c0 x1874671280630528/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827388.262729:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280630528/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827388.262739:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827388.262741:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827388.262744:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280630528:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827388.262750:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280630656:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827388.262764:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827388.262773:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827388/real 1787827388] req@ffff8ab3d0a43c00 x1874671280630656/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827388.262784:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280630656/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827388.262793:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827388.262795:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827388.262799:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280630656:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827389.286074:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827390.310051:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827391.334228:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827392.358061:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827393.381973:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827393.381981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827393.381985:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827393.381994:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1943 00000100:00080000:2.0:1787827393.382003:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827393.382009:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787827393.382016:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827393.382068:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280630784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827393.382099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827393.382103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827393.382105:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827393.382109:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1943 00000100:00080000:2.0:1787827393.382124:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827393.382129:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827393.382140:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827393.382172:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280630912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827393.382183:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827393.382187:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787827393.382192:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280630784:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827393.382192:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827393.382203:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1943 00000100:00080000:2.0:1787827393.382207:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827393.382210:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827393.382216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827393.382229:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280631040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827393.382245:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827393.382250:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827393.382253:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827393.382261:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1943 00000800:00000100:1.0:1787827393.382265:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827393.382269:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827393.382280:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827393.382288:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827393.382307:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280631168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827393.382316:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827393/real 1787827393] req@ffff8ab3f043ad00 x1874671280630784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827393.382321:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827393.382325:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827393.382327:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827393.382332:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1943 00000100:00080000:2.0:1787827393.382338:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827393.382343:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827393.382353:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827393.382362:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280630784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827393.382369:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280631296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827393.382402:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827393.382412:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827393.382419:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280630784:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827393.382448:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280630912:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827393.382476:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827393.382499:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827393/real 1787827393] req@ffff8ab3f0438780 x1874671280630912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827393.382521:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280630912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827393.382535:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827393.382540:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827393.382561:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280630912:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827393.382569:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280631040:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827393.382594:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827393.382608:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827393/real 1787827393] req@ffff8ab3f0438f00 x1874671280631040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827393.382646:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280631040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827393.382655:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827393.382658:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827393.382661:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280631040:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827393.382668:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280631168:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827393.382715:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827393.382732:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827393/real 1787827393] req@ffff8ab3f043a580 x1874671280631168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827393.382753:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280631168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827393.382768:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827393.382774:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827393.382780:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280631168:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827393.382816:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280631296:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827393.382855:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827393.382876:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827393/real 1787827393] req@ffff8ab3f043b0c0 x1874671280631296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827393.382917:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280631296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827393.382927:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827393.382931:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827393.382934:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280631296:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827394.406051:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827395.430085:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827396.453988:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827397.478078:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827398.502056:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827398.502062:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827398.502073:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827398.502077:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827398.502090:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1948 00000100:00080000:3.0:1787827398.502102:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827398.502112:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827398.502167:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280631424/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827398.502202:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827398.502207:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827398.502209:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827398.502213:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1948 00000100:00080000:3.0:1787827398.502217:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827398.502229:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827398.502242:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827398.502256:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280631552/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827398.502270:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827398.502281:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787827398.502282:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280631424:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827398.502289:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827398.502300:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1948 00000100:00080000:3.0:1787827398.502307:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827398.502315:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827398.502326:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827398.502350:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280631680/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827398.502370:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827398.502373:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827398.502383:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827398.502388:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827398.502397:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1948 00000100:00080000:3.0:1787827398.502404:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827398.502411:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827398.502420:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827398.502437:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280631808/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827398.502449:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827398.502453:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827398.502457:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827398.502466:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1948 00000100:00080000:3.0:1787827398.502474:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827398.502481:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827398/real 1787827398] req@ffff8ab3d0a40000 x1874671280631424/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827398.502488:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827398.502505:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827398.502520:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280631424/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827398.502534:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280631936/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827398.502540:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827398.502562:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827398.502572:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280631424:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827398.502615:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280631552:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827398.502646:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827398.502671:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827398/real 1787827398] req@ffff8ab3d0a43480 x1874671280631552/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827398.502695:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280631552/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827398.502705:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827398.502709:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827398.502713:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280631552:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827398.502719:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280631680:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827398.502737:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827398.502769:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827398/real 1787827398] req@ffff8ab3d0a43840 x1874671280631680/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827398.502787:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280631680/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827398.502797:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827398.502808:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827398.502812:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280631680:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827398.502819:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280631808:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827398.502844:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827398.502856:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827398/real 1787827398] req@ffff8ab3d0a430c0 x1874671280631808/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827398.502869:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280631808/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827398.502916:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827398.502921:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827398.502925:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280631808:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827398.502932:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280631936:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827398.502953:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827398.502977:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827398/real 1787827398] req@ffff8ab3d0a43c00 x1874671280631936/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827398.502990:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280631936/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827398.503001:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827398.503005:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827398.503008:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280631936:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827399.526022:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827400.550042:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827401.574117:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827402.597998:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827403.622094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827403.622115:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827403.622123:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827403.622171:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1953 00000100:00080000:3.0:1787827403.622194:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827403.622223:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827403.622262:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827403.622312:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280632064/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827403.622384:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827403.622391:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827403.622394:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827403.622403:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1953 00000100:00080000:3.0:1787827403.622411:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827403.622421:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827403.622438:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827403.622442:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280632064:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827403.622466:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280632192/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827403.622528:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827403.622536:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827403.622549:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827403.622578:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1953 00000100:00080000:3.0:1787827403.622593:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827403.622603:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827403.622611:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827403.622634:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827403.622662:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280632320/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827403.622683:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827403.622691:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827403.622696:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827403.622705:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1953 00000100:00100000:1.0:1787827403.622735:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827403/real 1787827403] req@ffff8ab3d0a412c0 x1874671280632064/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827403.622746:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827403.622767:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827403.622782:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787827403.622790:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280632064/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827403.622821:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827403.622827:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280632448/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827403.622834:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827403.622849:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280632064:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827403.622871:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827403.622879:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827403.622883:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827403.622937:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280632192:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787827403.622948:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1953 00000100:00080000:3.0:1787827403.622976:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827403.622996:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827403.623006:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787827403.623023:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827403.623032:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280632576/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827403.623066:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827403/real 1787827403] req@ffff8ab3d0a421c0 x1874671280632192/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827403.623088:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280632192/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827403.623098:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827403.623102:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827403.623107:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280632192:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827403.623115:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280632320:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827403.623140:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827403.623174:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827403/real 1787827403] req@ffff8ab3d0a403c0 x1874671280632320/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827403.623188:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280632320/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827403.623198:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827403.623202:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827403.623206:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280632320:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827403.623213:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280632448:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827403.623245:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827403.623257:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827403/real 1787827403] req@ffff8ab3d0a41e00 x1874671280632448/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827403.623271:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280632448/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827403.623283:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827403.623286:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827403.623290:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280632448:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827403.623311:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280632576:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827403.623331:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827403.623342:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827403/real 1787827403] req@ffff8ab3d0a412c0 x1874671280632576/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827403.623356:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280632576/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827403.623369:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827403.623372:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827403.623376:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280632576:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827404.646057:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827405.670122:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827406.694125:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827407.718113:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827408.741992:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827408.742019:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1787827408.742027:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827408.742033:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827408.742045:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1958 00000100:00080000:0.0:1787827408.742055:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827408.742063:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827408.742113:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280632704/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827408.742139:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827408.742142:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827408.742143:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827408.742145:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1958 00000100:00080000:0.0:1787827408.742148:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827408.742151:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827408.742159:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827408.742169:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280632832/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827408.742178:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827408.742180:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280632704:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827408.742181:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827408.742185:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827408.742191:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1958 00000100:00080000:0.0:1787827408.742193:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827408.742196:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827408.742205:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827408.742215:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280632960/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827408.742222:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827408.742224:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827408.742226:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827408.742228:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1958 00000100:00080000:0.0:1787827408.742231:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827408.742234:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827408.742238:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787827408.742242:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827408.742250:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280633088/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827408.742258:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827408.742261:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827408.742263:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827408.742265:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1958 00000100:00080000:0.0:1787827408.742267:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827408.742270:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827408.742274:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827408.742283:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280633216/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827408.742290:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827408/real 1787827408] req@ffff8ab3f075ed00 x1874671280632704/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827408.742308:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280632704/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827408.742319:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827408.742324:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827408.742329:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280632704:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827408.742353:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280632832:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827408.742376:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827408.742387:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827408/real 1787827408] req@ffff8ab3f075da40 x1874671280632832/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827408.742395:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280632832/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827408.742401:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827408.742403:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827408.742405:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280632832:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827408.742409:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280632960:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827408.742420:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827408.742425:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827408/real 1787827408] req@ffff8ab3f075cb40 x1874671280632960/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827408.742433:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280632960/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827408.742448:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827408.742451:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827408.742453:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280632960:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827408.742456:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280633088:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827408.742468:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827408.742488:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827408/real 1787827408] req@ffff8ab3f075e580 x1874671280633088/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827408.742496:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280633088/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827408.742502:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827408.742504:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827408.742506:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280633088:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827408.742510:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280633216:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827408.742519:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827408.742524:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827408/real 1787827408] req@ffff8ab3f075cf00 x1874671280633216/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827408.742531:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280633216/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827408.742537:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827408.742539:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827408.742541:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280633216:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827409.766116:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827410.790059:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827411.814094:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827412.837987:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827413.862038:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827413.862051:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827413.862057:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827413.862071:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1963 00000100:00080000:0.0:1787827413.862084:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827413.862094:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827413.862121:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827413.862145:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280633344/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827413.862182:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827413.862187:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827413.862190:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827413.862194:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1963 00000100:00080000:0.0:1787827413.862199:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827413.862205:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827413.862219:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827413.862238:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280633472/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827413.862242:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280633344:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827413.862269:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827413.862273:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827413.862276:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827413.862283:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1963 00000100:00080000:0.0:1787827413.862287:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827413.862292:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827413.862299:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827413.862314:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280633600/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827413.862327:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827413.862332:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827413.862338:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827413.862339:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787827413.862346:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1963 00000100:00080000:0.0:1787827413.862354:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827413.862360:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827413.862367:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827413.862384:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280633728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827413.862397:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827413.862402:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827413.862404:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827413.862413:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1963 00000100:00100000:1.0:1787827413.862415:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827413/real 1787827413] req@ffff8ab3f075f480 x1874671280633344/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827413.862418:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827413.862430:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827413.862444:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827413.862463:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280633344/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827413.862466:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280633856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827413.862521:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827413.862531:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827413.862538:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280633344:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827413.862569:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280633472:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827413.862595:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827413.862611:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827413/real 1787827413] req@ffff8ab3f075f0c0 x1874671280633472/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827413.862627:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280633472/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827413.862638:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827413.862665:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827413.862670:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280633472:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827413.862678:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280633600:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827413.862701:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827413.862714:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827413/real 1787827413] req@ffff8ab3f075f840 x1874671280633600/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827413.862729:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280633600/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827413.862738:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827413.862743:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827413.862746:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280633600:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827413.862753:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280633728:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827413.862771:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827413.862782:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827413/real 1787827413] req@ffff8ab3f075e940 x1874671280633728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827413.862795:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280633728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827413.862806:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827413.862809:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827413.862813:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280633728:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827413.862820:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280633856:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827413.862837:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827413.862847:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827413/real 1787827413] req@ffff8ab3f075c000 x1874671280633856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827413.862869:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280633856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827413.862880:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827413.862883:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827413.862920:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280633856:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827414.886111:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827415.910101:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827416.934093:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827417.958152:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827418.982088:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827418.982099:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827418.982104:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827418.982116:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1968 00010000:00010000:1.0:1787827418.982147:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827418.982148:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827418.982162:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827418.982220:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280633984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827418.982254:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827418.982258:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827418.982260:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827418.982264:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1968 00000100:00080000:0.0:1787827418.982267:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827418.982273:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827418.982285:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827418.982302:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280634112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827418.982308:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280633984:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827418.982324:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827418.982329:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827418.982331:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827418.982336:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1968 00000100:00080000:0.0:1787827418.982338:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827418.982342:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827418.982347:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827418.982359:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280634240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827418.982376:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827418.982380:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827418.982382:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827418.982385:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1968 00000100:00080000:0.0:1787827418.982397:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827418.982402:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827418.982407:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827418.982419:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280634368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827418.982428:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827418.982432:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827418.982434:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827418.982438:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1968 00000100:00080000:0.0:1787827418.982442:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827418.982447:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827418.982451:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827418.982464:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280634496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827418.982476:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280634112:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827418.982524:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280634240:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827418.982544:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280634368:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827418.982595:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280634496:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827420.006210:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827421.030044:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827422.054022:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787827422.054116:0:10944:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787827422.054148:0:10944:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827422.054161:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827422.054188:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827422.054195:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827422.054203:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827422.054211:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787827422.054223:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827418/real 1787827422] req@ffff8ab3f075c000 x1874671280634112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827422.054245:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280634112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827422.054254:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827422.054257:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827422.054261:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280634112:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827422.054267:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827418/real 1787827422] req@ffff8ab3f075e940 x1874671280634240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827422.054279:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280634240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827422.054287:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827422.054291:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827422.054295:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280634240:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827422.054301:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827418/real 1787827422] req@ffff8ab3f075f840 x1874671280634368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827422.054313:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280634368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827422.054323:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827422.054326:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827422.054329:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280634368:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827422.054336:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827418/real 1787827422] req@ffff8ab3f075de00 x1874671280633984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827422.054349:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280633984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827422.054357:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827422.054361:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827422.054365:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280633984:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827422.054392:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827418/real 1787827422] req@ffff8ab3f075f0c0 x1874671280634496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827422.054406:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280634496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827422.054416:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827422.054420:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827422.054423:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280634496:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827423.078079:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827424.102123:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827424.102136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827424.102142:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1787827424.102155:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827424.102158:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1973 00000100:00080000:2.0:1787827424.102173:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827424.102182:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827424.102239:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280634624/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827424.102272:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827424.102276:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827424.102278:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827424.102281:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1973 00000100:00080000:2.0:1787827424.102285:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827424.102288:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827424.102298:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827424.102311:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438000 x1874671280634752/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827424.102322:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827424.102325:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827424.102327:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827424.102331:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1973 00000100:00080000:2.0:1787827424.102333:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827424.102337:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827424.102343:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827424.102346:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280634624:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827424.102356:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280634880/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827424.102374:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827424.102379:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827424.102382:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827424.102391:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1973 00000100:00080000:2.0:1787827424.102397:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827424.102420:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827424.102429:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787827424.102440:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827424.102461:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280635008/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827424.102495:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827424/real 1787827424] req@ffff8ab3f0438b40 x1874671280634624/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827424.102498:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827424.102515:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827424.102521:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827424.102532:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1973 00000100:00100000:1.0:1787827424.102537:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280634624/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827424.102541:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827424.102554:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787827424.102590:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787827424.102604:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787827424.102606:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827424.102617:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280634624:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827424.102636:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280634752:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827424.102637:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280635136/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827424.102687:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827424.102743:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827424/real 1787827424] req@ffff8ab3f0438000 x1874671280634752/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827424.102770:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438000 x1874671280634752/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827424.102781:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827424.102803:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827424.102817:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280634752:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827424.102850:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280634880:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827424.102874:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827424.102913:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827424/real 1787827424] req@ffff8ab3f043bc00 x1874671280634880/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827424.102927:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280634880/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827424.102937:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827424.102941:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827424.102944:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280634880:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827424.102951:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280635008:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827424.102978:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827424.102991:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827424/real 1787827424] req@ffff8ab3f04383c0 x1874671280635008/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827424.103003:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280635008/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827424.103019:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827424.103022:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827424.103025:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280635008:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827424.103031:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280635136:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827424.103053:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827424.103063:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827424/real 1787827424] req@ffff8ab3f04392c0 x1874671280635136/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827424.103073:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280635136/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827424.103083:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827424.103086:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827424.103089:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280635136:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827425.126048:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827426.150037:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827427.174104:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827428.198112:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827429.221997:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827429.222008:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1787827429.222011:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827429.222014:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827429.222025:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1978 00000100:00080000:3.0:1787827429.222036:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827429.222044:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827429.222083:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280635264/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827429.222118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827429.222122:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827429.222125:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827429.222133:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1978 00000100:00080000:3.0:1787827429.222139:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827429.222146:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827429.222147:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280635264:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787827429.222159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827429.222175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280635392/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827429.222188:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827429.222194:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827429.222197:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827429.222202:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1978 00000800:00000100:1.0:1787827429.222205:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827429.222209:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827429.222217:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827429.222223:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827429.222237:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280635520/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827429.222248:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827429.222252:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827429.222254:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827429.222256:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827429/real 1787827429] req@ffff8ab3d0a43c00 x1874671280635264/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827429.222262:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1978 00000100:00080000:3.0:1787827429.222272:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827429.222280:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827429.222281:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280635264/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827429.222290:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787827429.222303:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827429.222306:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827429.222311:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280635648/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827429.222312:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280635264:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827429.222327:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827429.222331:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827429.222333:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827429.222337:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280635392:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787827429.222341:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1978 00000100:00080000:3.0:1787827429.222347:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827429.222352:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827429.222359:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787827429.222363:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827429.222376:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280635776/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827429.222384:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827429/real 1787827429] req@ffff8ab3d0a430c0 x1874671280635392/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827429.222403:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280635392/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827429.222413:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827429.222416:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827429.222419:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280635392:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827429.222423:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280635520:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827429.222439:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827429.222447:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827429/real 1787827429] req@ffff8ab3d0a43840 x1874671280635520/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827429.222454:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280635520/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827429.222460:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827429.222462:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827429.222464:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280635520:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827429.222468:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280635648:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827429.222478:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827429.222483:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827429/real 1787827429] req@ffff8ab3d0a43480 x1874671280635648/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827429.222490:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280635648/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827429.222507:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827429.222509:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827429.222511:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280635648:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827429.222522:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280635776:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827429.222545:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827429.222555:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827429/real 1787827429] req@ffff8ab3d0a40000 x1874671280635776/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827429.222563:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280635776/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827429.222569:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827429.222571:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827429.222574:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280635776:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827430.246088:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827431.269977:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827432.294075:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827433.318025:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827434.342010:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827434.342024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827434.342030:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827434.342045:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1983 00000100:00080000:3.0:1787827434.342055:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827434.342063:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827434.342106:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280635904/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827434.342137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827434.342142:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00010000:00010000:0.0:1787827434.342145:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827434.342146:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827434.342156:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1983 00000100:00080000:3.0:1787827434.342163:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827434.342170:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827434.342184:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827434.342195:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280635904:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827434.342206:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280636032/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827434.342232:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827434.342235:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827434.342237:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827434.342241:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1983 00000100:00080000:3.0:1787827434.342259:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827434.342264:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827434.342270:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827434.342277:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827434.342289:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280636160/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827434.342300:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827434.342303:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827434.342305:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827434.342309:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1983 00000100:00080000:3.0:1787827434.342312:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827434.342316:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827434.342321:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827434.342331:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280636288/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827434.342340:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827434.342344:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827434.342345:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827434.342350:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1983 00000100:00080000:3.0:1787827434.342353:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827434.342358:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827434.342363:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827434.342379:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280636416/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827434.342382:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827434/real 1787827434] req@ffff8ab3d0a40000 x1874671280635904/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827434.342454:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280635904/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827434.342471:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827434.342478:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827434.342484:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280635904:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827434.342537:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280636032:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827434.342574:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827434.342593:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827434/real 1787827434] req@ffff8ab3d0a43480 x1874671280636032/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827434.342614:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280636032/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827434.342626:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827434.342631:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827434.342636:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280636032:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827434.342646:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280636160:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827434.342676:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827434.342693:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827434/real 1787827434] req@ffff8ab3d0a43840 x1874671280636160/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827434.342737:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280636160/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827434.342750:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827434.342754:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827434.342759:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280636160:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827434.342768:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280636288:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827434.342798:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827434.342816:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827434/real 1787827434] req@ffff8ab3d0a430c0 x1874671280636288/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827434.342834:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280636288/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827434.342847:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827434.342852:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827434.342857:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280636288:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827434.342866:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280636416:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827434.342929:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827434.342948:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827434/real 1787827434] req@ffff8ab3d0a43c00 x1874671280636416/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827434.342965:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280636416/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827434.342988:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827434.342993:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827434.342998:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280636416:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827435.366003:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827436.390076:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827437.413984:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827438.438070:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827439.462028:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827439.462030:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827439.462046:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827439.462050:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827439.462062:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1989 00000100:00080000:0.0:1787827439.462074:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827439.462081:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827439.462133:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280636544/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827439.462167:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827439.462172:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827439.462174:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827439.462178:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1989 00000100:00080000:0.0:1787827439.462184:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827439.462189:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827439.462203:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827439.462208:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280636544:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827439.462220:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280636672/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827439.462234:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827439.462238:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827439.462240:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827439.462246:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1989 00000100:00080000:0.0:1787827439.462250:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827439.462255:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827439.462262:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827439.462272:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280636800/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827439.462282:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827439.462286:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787827439.462289:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787827439.462291:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827439.462301:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1989 00000100:00080000:0.0:1787827439.462306:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827439.462312:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827439.462318:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827439.462334:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280636928/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827439.462340:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827439/real 1787827439] req@ffff8ab3f075f480 x1874671280636544/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827439.462359:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827439.462366:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827439.462368:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827439.462374:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280636544/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827439.462380:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1989 00000100:00080000:0.0:1787827439.462393:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827439.462396:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827439.462403:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787827439.462404:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827439.462413:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280636544:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787827439.462417:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827439.462435:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280637056/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827439.462440:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280636672:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827439.462496:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827439.462514:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827439/real 1787827439] req@ffff8ab3f075cf00 x1874671280636672/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827439.462522:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280636672/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827439.462533:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827439.462535:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827439.462538:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280636672:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827439.462544:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280636800:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827439.462563:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827439.462574:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827439/real 1787827439] req@ffff8ab3f075e580 x1874671280636800/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827439.462587:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280636800/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827439.462595:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827439.462599:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827439.462603:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280636800:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827439.462609:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280636928:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827439.462630:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827439.462643:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827439/real 1787827439] req@ffff8ab3f075cb40 x1874671280636928/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827439.462655:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280636928/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827439.462663:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827439.462666:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827439.462668:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280636928:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827439.462682:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280637056:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827439.462705:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827439.462717:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827439/real 1787827439] req@ffff8ab3f075da40 x1874671280637056/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827439.462729:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280637056/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827439.462740:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827439.462744:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827439.462747:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280637056:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827440.486095:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827441.509978:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827442.534173:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827443.558057:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827444.582053:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827444.582065:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827444.582070:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1787827444.582078:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827444.582083:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1994 00000100:00080000:3.0:1787827444.582096:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827444.582103:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827444.582166:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280637184/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827444.582195:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827444.582199:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827444.582201:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827444.582204:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1994 00000100:00080000:3.0:1787827444.582208:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827444.582212:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827444.582223:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827444.582234:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280637184:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827444.582237:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280637312/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827444.582250:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827444.582253:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827444.582255:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827444.582258:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1994 00000100:00080000:3.0:1787827444.582261:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827444.582264:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827444.582269:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827444.582281:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280637440/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827444.582289:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827444.582292:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827444.582293:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827444.582296:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1994 00000100:00080000:3.0:1787827444.582308:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827444.582312:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827444.582316:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787827444.582322:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827444.582332:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280637568/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827444.582341:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827444.582344:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827444.582346:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827444.582350:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1994 00000100:00080000:3.0:1787827444.582353:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827444.582356:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827444.582360:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827444.582370:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280637696/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827444.582388:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827444/real 1787827444] req@ffff8ab3d0a412c0 x1874671280637184/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827444.582407:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280637184/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827444.582418:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827444.582422:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827444.582425:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280637184:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827444.582469:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280637312:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827444.582492:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827444.582503:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827444/real 1787827444] req@ffff8ab3d0a41e00 x1874671280637312/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827444.582523:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280637312/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827444.582530:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827444.582533:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827444.582536:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280637312:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827444.582541:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280637440:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827444.582555:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827444.582563:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827444/real 1787827444] req@ffff8ab3d0a403c0 x1874671280637440/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827444.582572:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280637440/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827444.582578:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827444.582580:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827444.582583:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280637440:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827444.582587:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280637568:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827444.582600:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827444.582606:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827444/real 1787827444] req@ffff8ab3d0a421c0 x1874671280637568/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827444.582621:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280637568/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827444.582628:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827444.582630:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827444.582632:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280637568:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827444.582637:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280637696:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827444.582649:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827444.582656:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827444/real 1787827444] req@ffff8ab3d0a43c00 x1874671280637696/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827444.582664:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280637696/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827444.582672:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827444.582674:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827444.582677:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280637696:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827445.606020:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827446.630095:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827447.654000:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827448.678065:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827449.701964:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827449.701972:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827449.701976:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827449.701985:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 1999 00010000:00010000:1.0:1787827449.701990:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827449.701994:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827449.702002:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827449.702035:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280637824/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827449.702057:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827449.702059:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827449.702061:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827449.702063:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1999 00000100:00080000:3.0:1787827449.702066:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827449.702069:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827449.702078:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827449.702087:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280637824:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827449.702089:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280637952/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827449.702101:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827449.702103:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827449.702104:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827449.702107:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1999 00000100:00080000:3.0:1787827449.702110:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827449.702112:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827449.702116:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827449.702124:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280638080/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827449.702135:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827449.702137:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827449.702139:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827449.702141:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1999 00000100:00080000:3.0:1787827449.702144:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827449.702148:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827449.702151:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827449.702156:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827449.702165:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280638208/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827449.702172:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827449.702174:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827449.702176:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827449.702179:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 1999 00000100:00080000:3.0:1787827449.702182:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827449.702184:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827449.702188:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827449.702195:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827449/real 1787827449] req@ffff8ab3d0a430c0 x1874671280637824/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827449.702198:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280638336/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827449.702251:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280637824/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827449.702259:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827449.702262:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827449.702265:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280637824:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827449.702283:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280637952:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827449.702301:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827449.702321:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827449/real 1787827449] req@ffff8ab3d0a43840 x1874671280637952/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827449.702335:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280637952/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827449.702344:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827449.702347:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827449.702351:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280637952:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827449.702358:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280638080:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827449.702389:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827449.702412:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827449/real 1787827449] req@ffff8ab3d0a43480 x1874671280638080/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827449.702425:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280638080/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827449.702440:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827449.702443:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827449.702447:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280638080:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827449.702454:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280638208:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827449.702489:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827449.702502:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827449/real 1787827449] req@ffff8ab3d0a40000 x1874671280638208/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827449.702515:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280638208/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827449.702526:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827449.702529:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827449.702533:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280638208:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827449.702540:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280638336:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827449.702561:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827449.702573:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827449/real 1787827449] req@ffff8ab3d0a43c00 x1874671280638336/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827449.702586:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280638336/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827449.702596:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827449.702599:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827449.702603:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280638336:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827450.726059:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827451.749985:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827452.774061:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827453.798063:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827454.822031:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827454.822039:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827454.822043:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827454.822052:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2004 00000100:00080000:3.0:1787827454.822062:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827454.822068:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827454.822102:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280638464/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827454.822124:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827454.822127:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827454.822128:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827454.822132:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2004 00010000:00010000:0.0:1787827454.822134:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827454.822135:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827454.822140:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827454.822149:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827454.822157:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280638592/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827454.822162:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280638464:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827454.822174:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827454.822177:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827454.822178:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827454.822182:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2004 00000100:00080000:3.0:1787827454.822184:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827454.822187:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827454.822190:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827454.822200:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280638720/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827454.822206:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827454.822209:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827454.822210:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827454.822213:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2004 00000100:00080000:3.0:1787827454.822216:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827454.822218:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827454.822222:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787827454.822226:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827454.822235:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280638848/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827454.822243:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827454.822245:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827454.822246:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827454.822249:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2004 00000100:00080000:3.0:1787827454.822268:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827454.822271:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827454.822275:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827454.822287:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827454/real 1787827454] req@ffff8ab3d0a421c0 x1874671280638464/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827454.822287:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280638976/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827454.822330:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280638464/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827454.822340:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827454.822343:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827454.822346:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280638464:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827454.822379:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280638592:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827454.822413:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827454.822422:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827454/real 1787827454] req@ffff8ab3d0a403c0 x1874671280638592/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827454.822431:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280638592/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827454.822437:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827454.822439:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827454.822441:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280638592:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827454.822445:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280638720:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827454.822455:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827454.822461:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827454/real 1787827454] req@ffff8ab3d0a41e00 x1874671280638720/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827454.822468:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280638720/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827454.822473:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827454.822474:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827454.822476:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280638720:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827454.822480:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280638848:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827454.822499:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827454.822505:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827454/real 1787827454] req@ffff8ab3d0a412c0 x1874671280638848/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827454.822512:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280638848/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827454.822518:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827454.822520:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827454.822521:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280638848:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827454.822525:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280638976:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827454.822533:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827454.822539:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827454/real 1787827454] req@ffff8ab3d0a43c00 x1874671280638976/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827454.822560:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280638976/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827454.822566:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827454.822567:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827454.822569:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280638976:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827455.846008:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827456.870106:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827457.893995:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827458.918078:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827459.942011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827459.942020:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827459.942024:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1787827459.942026:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827459.942033:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2009 00000100:00080000:2.0:1787827459.942043:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827459.942049:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827459.942088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280639104/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827459.942115:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827459.942119:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827459.942121:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827459.942124:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2009 00000100:00080000:2.0:1787827459.942128:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827459.942131:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827459.942141:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827459.942154:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280639232/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827459.942156:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280639104:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827459.942170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827459.942175:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827459.942176:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827459.942179:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2009 00000100:00080000:2.0:1787827459.942181:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827459.942183:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827459.942188:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827459.942195:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280639360/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827459.942202:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827459.942204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827459.942206:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827459.942208:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2009 00000100:00080000:2.0:1787827459.942218:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827459.942222:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827459.942225:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787827459.942231:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827459.942235:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280639488/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827459.942242:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827459.942245:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827459.942246:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827459.942250:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2009 00000100:00080000:2.0:1787827459.942255:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827459.942257:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827459.942261:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827459.942269:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280639616/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827459.942293:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827459/real 1787827459] req@ffff8ab3f0439a40 x1874671280639104/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827459.942309:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280639104/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827459.942317:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827459.942335:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827459.942338:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280639104:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827459.942358:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280639232:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827459.942376:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827459.942385:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827459/real 1787827459] req@ffff8ab3f0439680 x1874671280639232/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827459.942398:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280639232/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827459.942404:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827459.942406:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827459.942408:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280639232:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827459.942412:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280639360:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827459.942424:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827459.942434:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827459/real 1787827459] req@ffff8ab3f043a1c0 x1874671280639360/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827459.942452:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280639360/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827459.942461:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827459.942464:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827459.942468:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280639360:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827459.942475:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280639488:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827459.942493:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827459.942501:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827459/real 1787827459] req@ffff8ab3f043a940 x1874671280639488/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827459.942509:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280639488/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827459.942520:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827459.942522:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827459.942525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280639488:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827459.942528:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280639616:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827459.942538:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827459.942543:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827459/real 1787827459] req@ffff8ab3f043b480 x1874671280639616/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827459.942550:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280639616/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827459.942557:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827459.942558:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827459.942560:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280639616:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827460.966018:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827461.990030:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827463.014043:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827464.038018:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827465.061981:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827465.061989:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827465.061993:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827465.062009:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2014 00000100:00080000:3.0:1787827465.062018:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827465.062025:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787827465.062042:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827465.062060:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280639744/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827465.062080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827465.062083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827465.062084:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827465.062087:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2014 00000100:00080000:3.0:1787827465.062089:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827465.062092:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827465.062100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827465.062108:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280639872/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827465.062116:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827465.062118:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827465.062120:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827465.062124:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2014 00000100:00080000:3.0:1787827465.062126:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827465.062127:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280639744:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827465.062129:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827465.062134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827465.062149:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280640000/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827465.062156:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827465.062158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827465.062160:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827465.062162:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2014 00000100:00080000:3.0:1787827465.062165:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827465.062168:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827465.062172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827465.062180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280640128/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827465.062187:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827465.062189:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827465.062190:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827465.062194:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2014 00000800:00000100:1.0:1787827465.062197:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827465.062199:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827465.062203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827465.062207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827465.062214:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280640256/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827465.062259:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827465/real 1787827465] req@ffff8ab3d0a40000 x1874671280639744/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827465.062278:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280639744/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827465.062303:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827465.062309:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827465.062313:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280639744:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827465.062338:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280639872:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827465.062362:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827465.062377:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827465/real 1787827465] req@ffff8ab3d0a43480 x1874671280639872/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827465.062389:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280639872/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827465.062398:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827465.062401:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827465.062405:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280639872:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827465.062412:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280640000:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827465.062432:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827465.062444:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827465/real 1787827465] req@ffff8ab3d0a43840 x1874671280640000/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827465.062457:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280640000/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827465.062465:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827465.062468:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827465.062472:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280640000:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827465.062479:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280640128:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827465.062498:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827465.062508:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827465/real 1787827465] req@ffff8ab3d0a430c0 x1874671280640128/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827465.062520:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280640128/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827465.062531:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827465.062534:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827465.062537:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280640128:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827465.062543:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280640256:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827465.062558:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827465.062567:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827465/real 1787827465] req@ffff8ab3d0a43c00 x1874671280640256/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827465.062580:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280640256/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827465.062589:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827465.062592:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827465.062596:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280640256:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827466.086102:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827467.110049:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827468.133980:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827469.158045:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827470.182015:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827470.182027:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827470.182032:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827470.182043:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2019 00010000:00010000:1.0:1787827470.182050:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827470.182056:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827470.182068:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827470.182107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280640384/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827470.182141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827470.182148:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827470.182150:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827470.182154:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2019 00000100:00080000:3.0:1787827470.182158:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827470.182163:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827470.182173:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827470.182187:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280640512/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827470.182198:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280640384:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827470.182201:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827470.182208:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827470.182210:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827470.182216:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2019 00000100:00080000:3.0:1787827470.182220:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827470.182225:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827470.182231:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827470.182244:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280640640/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827470.182268:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827470.182274:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827470.182275:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827470.182279:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827470.182284:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2019 00000100:00080000:3.0:1787827470.182289:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827470.182293:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827470.182300:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827470.182314:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280640768/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827470.182324:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827470.182327:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827470.182329:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827470.182336:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827470/real 1787827470] req@ffff8ab3d0a412c0 x1874671280640384/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827470.182337:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2019 00000100:00080000:3.0:1787827470.182343:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827470.182351:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827470.182359:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827470.182366:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280640384/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827470.182373:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280640896/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827470.182387:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827470.182393:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827470.182397:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280640384:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827470.182418:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280640512:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827470.182437:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827470.182451:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827470/real 1787827470] req@ffff8ab3d0a41e00 x1874671280640512/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827470.182464:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280640512/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827470.182472:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827470.182475:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827470.182478:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280640512:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827470.182484:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280640640:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827470.182502:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827470.182513:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827470/real 1787827470] req@ffff8ab3d0a403c0 x1874671280640640/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827470.182525:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280640640/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827470.182538:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827470.182541:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827470.182544:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280640640:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827470.182551:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280640768:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827470.182568:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827470.182579:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827470/real 1787827470] req@ffff8ab3d0a421c0 x1874671280640768/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827470.182589:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280640768/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827470.182598:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827470.182601:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827470.182603:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280640768:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827470.182609:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280640896:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827470.182624:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827470.182633:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827470/real 1787827470] req@ffff8ab3d0a43c00 x1874671280640896/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827470.182644:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280640896/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827470.182654:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827470.182657:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827470.182661:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280640896:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827471.206028:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827472.229998:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827473.254060:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827474.278001:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827475.301993:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827475.302002:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827475.302006:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827475.302014:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2024 00000100:00080000:2.0:1787827475.302022:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827475.302028:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787827475.302060:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827475.302072:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280641024/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827475.302092:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827475.302095:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827475.302096:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827475.302099:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2024 00000100:00080000:2.0:1787827475.302101:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827475.302104:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827475.302112:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827475.302121:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280641152/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827475.302130:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280641024:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827475.302132:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827475.302138:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827475.302141:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827475.302145:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2024 00000100:00080000:2.0:1787827475.302147:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827475.302150:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827475.302153:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827475.302163:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280641280/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827475.302169:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827475.302172:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827475.302173:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827475.302176:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2024 00000100:00080000:2.0:1787827475.302190:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827475.302193:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827475.302196:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787827475.302201:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827475.302206:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280641408/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827475.302227:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827475.302230:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827475.302232:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827475.302237:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2024 00000100:00080000:2.0:1787827475.302242:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827475.302249:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827475.302255:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827475/real 1787827475] req@ffff8ab3f043b840 x1874671280641024/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827475.302255:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827475.302277:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280641536/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827475.302278:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280641024/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827475.302312:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827475.302320:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827475.302325:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280641024:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827475.302344:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280641152:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827475.302359:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827475.302368:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827475/real 1787827475] req@ffff8ab3f0439e00 x1874671280641152/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827475.302377:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280641152/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827475.302382:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827475.302385:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827475.302387:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280641152:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827475.302390:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280641280:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827475.302401:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827475.302406:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827475/real 1787827475] req@ffff8ab3f043b480 x1874671280641280/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827475.302414:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280641280/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827475.302418:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827475.302420:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827475.302422:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280641280:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827475.302425:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280641408:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827475.302435:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827475.302440:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827475/real 1787827475] req@ffff8ab3f043a940 x1874671280641408/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827475.302447:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280641408/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827475.302453:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827475.302455:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827475.302457:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280641408:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827475.302460:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280641536:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827475.302469:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827475.302474:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827475/real 1787827475] req@ffff8ab3f043a1c0 x1874671280641536/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827475.302486:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280641536/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827475.302491:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827475.302493:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827475.302495:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280641536:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827476.326021:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827477.350029:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827478.374019:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827479.398047:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827480.421978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1787827480.421987:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827480.421988:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827480.421992:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827480.422001:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2030 00000100:00080000:3.0:1787827480.422013:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827480.422020:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827480.422056:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280641664/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827480.422078:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827480.422082:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827480.422084:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827480.422088:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2030 00000100:00080000:3.0:1787827480.422093:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827480.422098:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827480.422108:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827480.422111:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280641664:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827480.422119:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280641792/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827480.422129:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827480.422132:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827480.422133:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827480.422137:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2030 00000100:00080000:3.0:1787827480.422140:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827480.422143:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827480.422147:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827480.422156:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280641920/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827480.422164:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827480.422167:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787827480.422168:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827480.422169:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827480.422173:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2030 00000100:00080000:3.0:1787827480.422190:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827480.422195:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827480.422200:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827480.422211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280642048/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827480.422225:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827480/real 1787827480] req@ffff8ab3d0a430c0 x1874671280641664/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827480.422226:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827480.422229:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827480.422232:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827480.422236:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2030 00000100:00080000:3.0:1787827480.422239:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827480.422243:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280641664/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827480.422243:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827480.422247:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787827480.422253:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827480.422256:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280642176/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827480.422257:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827480.422261:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280641664:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827480.422283:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280641792:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827480.422306:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827480.422317:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827480/real 1787827480] req@ffff8ab3d0a43840 x1874671280641792/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827480.422330:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280641792/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827480.422336:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827480.422338:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827480.422340:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280641792:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827480.422344:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280641920:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827480.422355:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827480.422360:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827480/real 1787827480] req@ffff8ab3d0a43480 x1874671280641920/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827480.422367:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280641920/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827480.422372:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827480.422374:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827480.422376:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280641920:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827480.422379:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280642048:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827480.422389:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827480.422394:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827480/real 1787827480] req@ffff8ab3d0a40000 x1874671280642048/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827480.422401:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280642048/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827480.422412:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827480.422413:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827480.422416:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280642048:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827480.422419:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280642176:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827480.422428:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827480.422433:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827480/real 1787827480] req@ffff8ab3d0a43c00 x1874671280642176/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827480.422440:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280642176/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827480.422446:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827480.422448:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827480.422449:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280642176:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827481.446040:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827482.469988:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827483.494056:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827484.518000:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827485.542063:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827485.542073:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1787827485.542074:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827485.542079:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827485.542092:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2035 00000100:00080000:2.0:1787827485.542100:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827485.542106:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827485.542140:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280642304/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827485.542180:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827485.542183:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827485.542185:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827485.542187:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2035 00000100:00080000:2.0:1787827485.542190:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827485.542193:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827485.542202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827485.542212:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280642432/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827485.542221:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827485.542224:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827485.542225:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827485.542229:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2035 00000100:00080000:2.0:1787827485.542234:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827485.542237:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280642304:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827485.542238:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827485.542253:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827485.542271:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280642560/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827485.542281:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827485.542285:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827485.542287:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827485.542293:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2035 00000100:00080000:2.0:1787827485.542298:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827485.542303:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827485.542309:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827485.542318:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280642688/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827485.542325:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827485.542329:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827485.542331:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827485.542336:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2035 00000100:00080000:2.0:1787827485.542340:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827485.542345:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827485.542351:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827485.542369:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280642816/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827485.542405:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280642432:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827485.542455:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280642560:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827485.542485:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280642688:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827485.542521:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280642816:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827486.566034:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827487.590040:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827488.614000:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827488.614134:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787827488.614176:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827488.614189:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827488.614218:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827488.614224:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827488.614232:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827488.614241:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787827488.614261:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827485/real 1787827488] req@ffff8ab3f0439a40 x1874671280642432/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827488.614282:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280642432/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827488.614291:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827488.614294:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827488.614298:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280642432:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827488.614303:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827485/real 1787827488] req@ffff8ab3f04392c0 x1874671280642560/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827488.614310:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280642560/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827488.614315:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827488.614317:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827488.614319:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280642560:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827488.614323:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827485/real 1787827488] req@ffff8ab3f04383c0 x1874671280642688/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827488.614330:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280642688/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827488.614338:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827488.614340:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827488.614342:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280642688:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827488.614346:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827485/real 1787827488] req@ffff8ab3f043bc00 x1874671280642816/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827488.614352:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280642816/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827488.614358:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827488.614360:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827488.614362:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280642816:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827488.614365:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827485/real 1787827488] req@ffff8ab3f0439680 x1874671280642304/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827488.614372:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280642304/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827488.614377:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827488.614378:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827488.614380:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280642304:192.168.203.124@tcp:250:kworker.0 00010000:00010000:0.0:1787827489.638038:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827490.661961:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827490.661969:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827490.661973:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827490.661982:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2040 00010000:00010000:1.0:1787827490.661984:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827490.661991:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827490.662011:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827490.662042:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438000 x1874671280642944/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827490.662062:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827490.662065:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827490.662066:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827490.662069:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2040 00000100:00080000:2.0:1787827490.662071:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827490.662074:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827490.662083:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827490.662091:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280643072/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827490.662099:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827490.662103:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280642944:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827490.662103:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827490.662105:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827490.662125:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2040 00000100:00080000:2.0:1787827490.662128:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827490.662131:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827490.662136:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827490.662147:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280643200/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827490.662159:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827490.662162:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827490.662163:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827490.662166:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2040 00000100:00080000:2.0:1787827490.662169:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827490.662172:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827490.662175:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787827490.662180:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827490.662185:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280643328/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827490.662193:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827490.662196:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827490.662199:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827490.662203:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2040 00000100:00080000:2.0:1787827490.662206:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827490.662212:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827490.662221:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827490.662229:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827490/real 1787827490] req@ffff8ab3f0438000 x1874671280642944/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827490.662234:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280643456/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827490.662274:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438000 x1874671280642944/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827490.662287:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827490.662291:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827490.662296:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280642944:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827490.662318:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280643072:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827490.662349:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827490.662363:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827490/real 1787827490] req@ffff8ab3f0438b40 x1874671280643072/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827490.662375:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280643072/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827490.662383:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827490.662386:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827490.662389:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280643072:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827490.662395:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280643200:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827490.662412:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827490.662422:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827490/real 1787827490] req@ffff8ab3f043b0c0 x1874671280643200/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827490.662434:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280643200/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827490.662441:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827490.662444:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827490.662448:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280643200:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827490.662454:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280643328:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827490.662472:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827490.662482:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827490/real 1787827490] req@ffff8ab3f043a580 x1874671280643328/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827490.662493:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280643328/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827490.662503:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827490.662506:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827490.662510:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280643328:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827490.662516:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280643456:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827490.662540:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827490.662551:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827490/real 1787827490] req@ffff8ab3f0438f00 x1874671280643456/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827490.662563:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280643456/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827490.662572:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827490.662575:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827490.662579:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280643456:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827491.686072:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827492.710033:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827493.734208:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827494.758022:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827495.782116:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827495.782127:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827495.782131:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1787827495.782144:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827495.782144:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2045 00000100:00080000:2.0:1787827495.782163:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827495.782171:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827495.782207:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280643584/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827495.782254:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827495.782258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827495.782260:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827495.782264:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2045 00000100:00080000:2.0:1787827495.782269:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827495.782275:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827495.782287:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827495.782308:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280643584:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827495.782309:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280643712/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827495.782340:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827495.782361:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827495.782364:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827495.782371:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2045 00000100:00080000:2.0:1787827495.782375:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827495.782379:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827495.782385:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827495.782401:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280643840/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827495.782413:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827495.782419:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827495.782421:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827495.782425:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2045 00000100:00080000:2.0:1787827495.782429:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827495.782439:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787827495.782440:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787827495.782457:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827495.782476:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280643968/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827495.782494:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827495.782500:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827495.782520:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827495.782528:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2045 00000100:00080000:2.0:1787827495.782534:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827495.782544:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827495.782556:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827495.782561:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827495/real 1787827495] req@ffff8ab3f0438780 x1874671280643584/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827495.782576:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280644096/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827495.782593:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280643584/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827495.782618:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827495.782624:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827495.782629:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280643584:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827495.782659:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280643712:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827495.782684:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827495.782694:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827495/real 1787827495] req@ffff8ab3f043ad00 x1874671280643712/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827495.782704:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280643712/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827495.782710:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827495.782723:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827495.782726:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280643712:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827495.782730:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280643840:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827495.782742:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827495.782748:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827495/real 1787827495] req@ffff8ab3f0438f00 x1874671280643840/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827495.782762:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280643840/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827495.782768:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827495.782770:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827495.782772:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280643840:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827495.782776:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280643968:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827495.782807:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827495.782819:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827495/real 1787827495] req@ffff8ab3f043a580 x1874671280643968/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827495.782832:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280643968/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827495.782844:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827495.782848:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827495.782851:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280643968:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827495.782873:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280644096:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827495.782886:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827495.782893:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827495/real 1787827495] req@ffff8ab3f043b0c0 x1874671280644096/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827495.782900:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280644096/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827495.782913:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827495.782937:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827495.782940:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280644096:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827496.806050:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827497.830051:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827498.854038:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827499.878027:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827500.902003:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1787827500.902005:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827500.902015:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827500.902019:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827500.902028:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2050 00000100:00080000:3.0:1787827500.902050:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827500.902057:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827500.902117:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280644224/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827500.902152:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827500.902157:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827500.902159:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827500.902164:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2050 00000100:00080000:3.0:1787827500.902170:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827500.902178:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827500.902189:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280644224:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787827500.902192:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827500.902217:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280644352/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827500.902234:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827500.902237:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827500.902240:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827500.902244:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2050 00000100:00080000:3.0:1787827500.902248:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827500.902254:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827500.902258:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827500.902265:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827500.902280:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280644480/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827500.902291:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827500.902294:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827500.902296:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827500.902299:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2050 00000100:00080000:3.0:1787827500.902302:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827500.902306:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827500.902311:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787827500.902323:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827500/real 1787827500] req@ffff8ab3d0a421c0 x1874671280644224/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827500.902326:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280644608/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827500.902360:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280644224/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827500.902365:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827500.902393:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827500.902398:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827500.902404:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827500.902409:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2050 00000100:00080000:1.0:1787827500.902415:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787827500.902417:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827500.902423:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280644224:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827500.902429:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827500.902437:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827500.902450:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280644736/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827500.902460:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280644352:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827500.902514:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827500.902538:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827500/real 1787827500] req@ffff8ab3d0a403c0 x1874671280644352/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827500.902553:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280644352/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827500.902567:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827500.902571:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827500.902574:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280644352:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827500.902581:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280644480:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827500.902614:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827500.902622:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827500/real 1787827500] req@ffff8ab3d0a41e00 x1874671280644480/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827500.902630:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280644480/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827500.902636:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827500.902638:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827500.902640:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280644480:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827500.902644:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280644608:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827500.902655:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827500.902662:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827500/real 1787827500] req@ffff8ab3d0a412c0 x1874671280644608/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827500.902670:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280644608/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827500.902676:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827500.902678:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827500.902681:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280644608:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827500.902694:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280644736:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827500.902705:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827500.902712:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827500/real 1787827500] req@ffff8ab3d0a43c00 x1874671280644736/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827500.902720:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280644736/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827500.902727:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827500.902728:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827500.902730:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280644736:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827501.926124:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827502.949997:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827503.974033:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827504.998117:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827506.022011:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827506.022023:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827506.022026:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827506.022036:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2055 00000100:00080000:0.0:1787827506.022045:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827506.022069:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827506.022080:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827506.022106:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280644864/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827506.022128:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827506.022130:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827506.022132:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827506.022135:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2055 00000100:00080000:0.0:1787827506.022138:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827506.022141:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827506.022150:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827506.022162:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280644864:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827506.022163:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280644992/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827506.022178:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827506.022181:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827506.022182:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827506.022185:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2055 00000100:00080000:0.0:1787827506.022187:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827506.022190:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827506.022194:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827506.022202:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280645120/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827506.022209:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827506.022212:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827506.022214:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827506.022216:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2055 00000100:00080000:0.0:1787827506.022219:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827506.022222:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787827506.022227:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787827506.022229:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827506.022255:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280645248/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827506.022263:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827506.022266:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827506.022267:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827506.022270:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2055 00000100:00080000:0.0:1787827506.022273:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827506.022277:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827506.022282:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827506.022297:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280645376/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827506.022298:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827506/real 1787827506] req@ffff8ab3f075ed00 x1874671280644864/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827506.022331:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280644864/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827506.022367:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827506.022370:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827506.022373:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280644864:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827506.022393:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280644992:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827506.022411:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827506.022427:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827506/real 1787827506] req@ffff8ab3f075da40 x1874671280644992/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827506.022435:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280644992/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827506.022440:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827506.022443:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827506.022445:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280644992:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827506.022449:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280645120:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827506.022459:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827506.022464:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827506/real 1787827506] req@ffff8ab3f075cb40 x1874671280645120/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827506.022471:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280645120/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827506.022476:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827506.022478:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827506.022480:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280645120:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827506.022483:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280645248:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827506.022493:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827506.022498:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827506/real 1787827506] req@ffff8ab3f075e580 x1874671280645248/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827506.022524:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280645248/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827506.022531:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827506.022533:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827506.022535:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280645248:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827506.022539:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280645376:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827506.022550:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827506.022556:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827506/real 1787827506] req@ffff8ab3f075cf00 x1874671280645376/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827506.022563:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280645376/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827506.022569:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827506.022571:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827506.022573:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280645376:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827507.046091:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827508.070117:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827509.094077:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827510.118093:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827511.142043:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827511.142054:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827511.142058:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827511.142069:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2060 00000100:00080000:0.0:1787827511.142080:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827511.142088:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:1.0:1787827511.142090:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827511.142133:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280645504/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827511.142157:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827511.142160:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827511.142162:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827511.142164:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2060 00000100:00080000:0.0:1787827511.142167:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827511.142169:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827511.142178:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827511.142187:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280645632/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827511.142196:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827511.142202:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787827511.142203:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280645504:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827511.142205:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827511.142212:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2060 00000100:00080000:0.0:1787827511.142215:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827511.142223:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827511.142227:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827511.142236:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280645760/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827511.142244:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827511.142246:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827511.142248:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827511.142250:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2060 00000100:00080000:0.0:1787827511.142276:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827511.142279:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827511.142282:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787827511.142291:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827511.142292:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280645888/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827511.142301:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827511.142304:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827511.142306:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827511.142309:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2060 00000100:00080000:0.0:1787827511.142311:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827511.142314:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827511.142318:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827511.142325:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280646016/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827511.142359:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827511/real 1787827511] req@ffff8ab3f075f480 x1874671280645504/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827511.142400:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280645504/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827511.142412:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827511.142417:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827511.142422:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280645504:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827511.142449:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280645632:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827511.142473:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827511.142487:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827511/real 1787827511] req@ffff8ab3f075f0c0 x1874671280645632/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827511.142499:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280645632/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827511.142509:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827511.142512:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827511.142516:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280645632:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827511.142522:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280645760:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827511.142542:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827511.142554:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827511/real 1787827511] req@ffff8ab3f075de00 x1874671280645760/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827511.142565:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280645760/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827511.142580:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827511.142583:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827511.142586:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280645760:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827511.142592:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280645888:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827511.142609:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827511.142619:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827511/real 1787827511] req@ffff8ab3f075f840 x1874671280645888/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827511.142630:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280645888/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827511.142640:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827511.142643:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827511.142646:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280645888:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827511.142652:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280646016:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827511.142667:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827511.142676:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827511/real 1787827511] req@ffff8ab3f075e940 x1874671280646016/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827511.142686:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280646016/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827511.142695:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827511.142698:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827511.142702:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280646016:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827512.166068:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827513.190013:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827514.214059:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827515.238137:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827516.262010:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827516.262023:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827516.262029:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827516.262043:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2065 00010000:00010000:0.0:1787827516.262047:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827516.262056:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827516.262080:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827516.262120:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280646144/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827516.262147:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827516.262150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827516.262152:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827516.262156:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2065 00000100:00080000:2.0:1787827516.262160:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827516.262165:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827516.262175:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827516.262187:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280646144:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827516.262189:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438000 x1874671280646272/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827516.262204:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827516.262207:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827516.262208:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827516.262212:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2065 00000100:00080000:2.0:1787827516.262215:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827516.262219:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827516.262223:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827516.262235:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280646400/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827516.262249:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827516.262255:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827516.262256:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827516.262262:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827516.262266:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2065 00000100:00080000:2.0:1787827516.262271:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827516.262275:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827516.262280:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827516.262292:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280646528/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827516.262300:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827516.262303:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827516.262305:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827516.262309:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2065 00000100:00080000:2.0:1787827516.262313:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827516.262317:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827516/real 1787827516] req@ffff8ab3f0438b40 x1874671280646144/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827516.262320:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827516.262329:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827516.262347:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280646656/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827516.262350:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280646144/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827516.262381:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827516.262385:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827516.262389:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280646144:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827516.262430:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280646272:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827516.262452:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827516.262468:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827516/real 1787827516] req@ffff8ab3f0438000 x1874671280646272/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827516.262479:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438000 x1874671280646272/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827516.262485:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827516.262487:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827516.262489:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280646272:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827516.262493:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280646400:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827516.262526:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827516.262534:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827516/real 1787827516] req@ffff8ab3f0439680 x1874671280646400/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827516.262548:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280646400/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827516.262559:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827516.262563:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827516.262568:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280646400:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827516.262576:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280646528:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827516.262603:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827516.262620:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827516/real 1787827516] req@ffff8ab3f043bc00 x1874671280646528/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827516.262632:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280646528/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827516.262641:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827516.262644:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827516.262647:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280646528:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827516.262652:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280646656:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827516.262669:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827516.262677:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827516/real 1787827516] req@ffff8ab3f04383c0 x1874671280646656/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827516.262709:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280646656/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827516.262715:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827516.262717:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827516.262718:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280646656:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827517.286007:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827518.310075:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827519.334059:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827520.358082:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827521.382042:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827521.382065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827521.382075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827521.382082:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827521.382094:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2071 00000100:00080000:3.0:1787827521.382106:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827521.382127:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827521.382173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280646784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827521.382218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827521.382225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827521.382228:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827521.382234:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2071 00000100:00080000:3.0:1787827521.382241:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827521.382249:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827521.382263:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827521.382269:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280646784:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827521.382286:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280646912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827521.382315:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827521.382321:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827521.382324:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827521.382330:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2071 00000100:00080000:3.0:1787827521.382335:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827521.382346:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827521.382347:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827521.382358:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827521.382377:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280647040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827521.382399:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827521/real 1787827521] req@ffff8ab3d0a40000 x1874671280646784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827521.382405:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827521.382417:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827521.382420:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827521.382426:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280646784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827521.382433:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2071 00000100:00080000:3.0:1787827521.382455:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827521.382457:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827521.382466:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787827521.382469:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827521.382474:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280646784:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787827521.382482:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787827521.382506:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280646912:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827521.382506:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280647168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827521.382533:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827521.382541:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827521.382546:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827521.382552:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827521.382560:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827521/real 1787827521] req@ffff8ab3d0a43480 x1874671280646912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827521.382565:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2071 00000100:00080000:3.0:1787827521.382576:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827521.382589:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827521.382598:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280646912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827521.382600:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787827521.382615:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827521.382620:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827521.382627:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280646912:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827521.382629:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280647296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827521.382635:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280647040:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827521.382665:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827521.382676:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827521/real 1787827521] req@ffff8ab3d0a43840 x1874671280647040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827521.382688:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280647040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827521.382695:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827521.382697:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827521.382699:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280647040:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827521.382711:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280647168:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827521.382725:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827521.382733:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827521/real 1787827521] req@ffff8ab3d0a430c0 x1874671280647168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827521.382743:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280647168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827521.382768:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827521.382771:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827521.382774:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280647168:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827521.382779:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280647296:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827521.382791:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827521.382798:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827521/real 1787827521] req@ffff8ab3d0a40000 x1874671280647296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827521.382808:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280647296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827521.382816:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827521.382818:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827521.382821:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280647296:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827522.406101:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827523.429980:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827524.454131:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827525.478015:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827526.502057:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827526.502069:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827526.502075:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827526.502088:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2076 00000100:00080000:3.0:1787827526.502100:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827526.502124:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787827526.502164:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827526.502165:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280647424/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827526.502217:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827526.502222:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827526.502225:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827526.502230:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2076 00000100:00080000:3.0:1787827526.502234:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827526.502239:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827526.502245:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280647424:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787827526.502250:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827526.502266:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280647552/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827526.502278:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827526.502284:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827526.502286:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827526.502290:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2076 00000100:00080000:3.0:1787827526.502294:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827526.502299:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827526.502300:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827526.502307:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827526.502320:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280647680/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827526.502330:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827526.502334:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827526.502336:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827526.502340:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2076 00000100:00100000:1.0:1787827526.502343:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827526/real 1787827526] req@ffff8ab3d0a43c00 x1874671280647424/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827526.502345:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827526.502355:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827526.502360:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787827526.502362:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280647424/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827526.502373:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827526.502376:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280647808/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827526.502378:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827526.502382:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280647424:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827526.502395:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827526.502399:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787827526.502403:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280647552:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787827526.502403:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827526.502413:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2076 00000100:00080000:3.0:1787827526.502418:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827526.502423:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787827526.502427:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787827526.502432:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827526.502439:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827526/real 1787827526] req@ffff8ab3d0a412c0 x1874671280647552/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827526.502462:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280647936/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827526.502469:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280647552/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827526.502485:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827526.502489:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827526.502492:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280647552:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827526.502497:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280647680:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827526.502511:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827526.502518:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827526/real 1787827526] req@ffff8ab3d0a41e00 x1874671280647680/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827526.502525:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280647680/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827526.502530:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827526.502532:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827526.502534:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280647680:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827526.502538:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280647808:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827526.502548:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827526.502554:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827526/real 1787827526] req@ffff8ab3d0a403c0 x1874671280647808/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827526.502561:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280647808/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827526.502567:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827526.502569:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827526.502571:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280647808:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827526.502582:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280647936:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827526.502592:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827526.502597:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827526/real 1787827526] req@ffff8ab3d0a421c0 x1874671280647936/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827526.502605:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280647936/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827526.502611:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827526.502612:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827526.502614:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280647936:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827527.526038:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827528.550074:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827529.574075:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827530.598043:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827531.622081:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827531.622089:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827531.622098:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827531.622103:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827531.622115:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2081 00000100:00080000:0.0:1787827531.622127:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827531.622134:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827531.622193:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280648064/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827531.622220:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827531.622224:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827531.622226:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827531.622228:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2081 00000100:00080000:0.0:1787827531.622232:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827531.622235:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827531.622245:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827531.622260:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280648192/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827531.622282:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280648064:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827531.622293:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827531.622297:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827531.622298:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827531.622302:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2081 00000100:00080000:0.0:1787827531.622305:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827531.622308:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827531.622312:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827531.622323:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280648320/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827531.622331:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827531.622334:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827531.622336:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827531.622339:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2081 00000100:00080000:0.0:1787827531.622342:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827531.622345:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787827531.622351:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787827531.622352:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827531.622365:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280648448/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827531.622373:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827531.622376:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827531.622378:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827531.622382:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2081 00000100:00080000:0.0:1787827531.622386:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827531.622390:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827531.622395:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827531.622404:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827531/real 1787827531] req@ffff8ab3f075c000 x1874671280648064/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827531.622406:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280648576/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827531.622457:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280648064/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827531.622468:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827531.622472:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827531.622476:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280648064:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827531.622495:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280648192:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827531.622511:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827531.622521:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827531/real 1787827531] req@ffff8ab3f075e940 x1874671280648192/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827531.622531:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280648192/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827531.622538:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827531.622541:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827531.622543:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280648192:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827531.622548:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280648320:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827531.622559:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827531.622566:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827531/real 1787827531] req@ffff8ab3f075f840 x1874671280648320/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827531.622575:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280648320/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827531.622587:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827531.622589:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827531.622592:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280648320:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827531.622596:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280648448:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827531.622608:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827531.622615:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827531/real 1787827531] req@ffff8ab3f075de00 x1874671280648448/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827531.622642:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280648448/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827531.622651:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827531.622654:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827531.622657:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280648448:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827531.622663:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280648576:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827531.622678:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827531.622686:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827531/real 1787827531] req@ffff8ab3f075f0c0 x1874671280648576/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827531.622697:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280648576/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827531.622706:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827531.622709:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827531.622712:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280648576:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827532.646049:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827533.670029:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827534.694136:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827535.717985:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827536.742001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827536.742010:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827536.742014:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827536.742022:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2086 00000100:00080000:2.0:1787827536.742032:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827536.742038:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787827536.742045:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827536.742071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280648704/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827536.742092:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827536.742095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827536.742096:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827536.742099:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2086 00000100:00080000:2.0:1787827536.742101:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827536.742104:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827536.742113:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827536.742121:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280648832/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827536.742126:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280648704:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827536.742142:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827536.742148:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827536.742150:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827536.742153:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2086 00000100:00080000:2.0:1787827536.742157:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827536.742159:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827536.742163:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827536.742172:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280648960/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827536.742179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827536.742182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827536.742183:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827536.742186:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2086 00000100:00080000:2.0:1787827536.742189:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827536.742194:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827536.742196:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827536.742202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827536.742212:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280649088/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827536.742219:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827536.742222:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827536.742223:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827536.742226:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2086 00000100:00080000:2.0:1787827536.742229:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827536.742232:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827536.742236:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827536.742243:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827536/real 1787827536] req@ffff8ab3f04392c0 x1874671280648704/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827536.742245:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280649216/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827536.742290:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280648704/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827536.742300:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827536.742303:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827536.742314:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280648704:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827536.742332:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280648832:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827536.742350:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827536.742360:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827536/real 1787827536] req@ffff8ab3f0439a40 x1874671280648832/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827536.742368:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280648832/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827536.742374:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827536.742376:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827536.742378:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280648832:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827536.742382:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280648960:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827536.742393:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827536.742399:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827536/real 1787827536] req@ffff8ab3f043a1c0 x1874671280648960/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827536.742406:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280648960/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827536.742412:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827536.742413:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827536.742415:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280648960:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827536.742419:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280649088:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827536.742430:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827536.742435:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827536/real 1787827536] req@ffff8ab3f043a940 x1874671280649088/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827536.742447:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280649088/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827536.742458:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827536.742461:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827536.742465:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280649088:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827536.742471:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280649216:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827536.742491:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827536.742500:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827536/real 1787827536] req@ffff8ab3f043b480 x1874671280649216/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827536.742513:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280649216/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827536.742520:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827536.742521:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827536.742524:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280649216:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827537.766008:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827538.790203:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827539.814044:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827540.838049:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827541.862097:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827541.862118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827541.862129:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827541.862136:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827541.862149:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2091 00000100:00080000:3.0:1787827541.862161:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827541.862170:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827541.862231:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280649344/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827541.862281:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827541.862289:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827541.862293:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827541.862302:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2091 00000100:00080000:3.0:1787827541.862312:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827541.862319:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827541.862333:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827541.862336:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280649344:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827541.862359:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280649472/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827541.862378:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827541.862383:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827541.862386:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827541.862392:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2091 00000100:00080000:3.0:1787827541.862411:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827541.862417:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787827541.862426:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787827541.862428:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827541.862451:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280649600/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827541.862465:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827541.862469:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827541.862472:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827541.862480:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2091 00000100:00080000:3.0:1787827541.862486:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827541.862492:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827541.862500:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787827541.862502:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827541/real 1787827541] req@ffff8ab3d0a421c0 x1874671280649344/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827541.862520:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280649728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827541.862536:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827541.862538:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280649344/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827541.862549:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827541.862556:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827541.862570:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2091 00000100:00080000:1.0:1787827541.862574:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827541.862577:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827541.862591:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787827541.862595:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000008:00100000:3.0:1787827541.862599:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827541.862606:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280649344:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827541.862622:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280649856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827541.862647:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280649472:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827541.862682:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827541.862700:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827541/real 1787827541] req@ffff8ab3d0a403c0 x1874671280649472/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827541.862724:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280649472/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827541.862735:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827541.862738:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827541.862742:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280649472:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827541.862750:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280649600:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827541.862797:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827541.862810:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827541/real 1787827541] req@ffff8ab3d0a41e00 x1874671280649600/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827541.862824:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280649600/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827541.862834:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827541.862838:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827541.862842:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280649600:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827541.862850:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280649728:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827541.862881:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827541.862932:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827541/real 1787827541] req@ffff8ab3d0a412c0 x1874671280649728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827541.862947:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280649728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827541.862967:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827541.862971:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827541.862975:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280649728:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827541.862997:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280649856:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827541.863022:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827541.863034:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827541/real 1787827541] req@ffff8ab3d0a43c00 x1874671280649856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827541.863047:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280649856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827541.863059:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827541.863062:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827541.863065:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280649856:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827542.886102:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827543.910027:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827544.934067:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827545.958034:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827546.982013:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827546.982023:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827546.982027:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827546.982037:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2096 00000100:00080000:3.0:1787827546.982047:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827546.982054:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827546.982091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280649984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787827546.982094:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827546.982122:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827546.982126:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827546.982128:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827546.982131:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2096 00000100:00080000:3.0:1787827546.982135:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827546.982139:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827546.982148:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827546.982160:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280650112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827546.982169:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280649984:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827546.982176:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827546.982180:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827546.982182:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827546.982187:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2096 00000100:00080000:3.0:1787827546.982189:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827546.982192:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827546.982196:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827546.982208:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280650240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827546.982228:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827546.982231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827546.982232:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827546.982236:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2096 00000100:00080000:3.0:1787827546.982238:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827546.982242:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827546.982246:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827546.982257:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280650368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827546.982264:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827546.982267:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827546.982269:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827546.982272:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2096 00000100:00080000:3.0:1787827546.982287:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827546.982290:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827546.982293:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787827546.982303:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827546.982304:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280650496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827546.982388:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827546/real 1787827546] req@ffff8ab3d0a40000 x1874671280649984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827546.982415:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280649984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827546.982427:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827546.982432:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827546.982436:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280649984:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827546.982464:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280650112:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827546.982486:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827546.982517:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827546/real 1787827546] req@ffff8ab3d0a430c0 x1874671280650112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827546.982530:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280650112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827546.982538:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827546.982542:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827546.982546:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280650112:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827546.982551:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280650240:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827546.982568:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827546.982579:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827546/real 1787827546] req@ffff8ab3d0a43840 x1874671280650240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827546.982599:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280650240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827546.982625:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827546.982629:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827546.982634:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280650240:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827546.982642:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280650368:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827546.982666:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827546.982677:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827546/real 1787827546] req@ffff8ab3d0a43480 x1874671280650368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827546.982686:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280650368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827546.982695:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827546.982697:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827546.982700:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280650368:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827546.982705:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280650496:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827546.982738:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827546.982747:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827546/real 1787827546] req@ffff8ab3d0a43c00 x1874671280650496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827546.982756:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280650496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827546.982764:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827546.982767:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827546.982769:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280650496:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827548.006051:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827549.030100:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827550.054113:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827551.078166:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827552.102011:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827552.102020:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827552.102023:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827552.102031:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2101 00010000:00010000:1.0:1787827552.102033:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827552.102040:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827552.102047:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827552.102083:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280650624/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827552.102110:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827552.102115:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827552.102117:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827552.102122:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2101 00000100:00080000:0.0:1787827552.102127:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827552.102132:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827552.102142:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827552.102147:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280650624:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827552.102156:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280650752/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827552.102165:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827552.102167:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827552.102169:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827552.102171:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2101 00000100:00080000:0.0:1787827552.102174:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827552.102176:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827552.102180:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827552.102187:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280650880/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827552.102194:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827552.102196:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827552.102197:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827552.102200:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2101 00000100:00080000:0.0:1787827552.102202:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827552.102205:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827552.102208:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827552.102215:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280651008/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827552.102222:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827552.102224:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827552.102225:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827552.102227:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2101 00000100:00080000:0.0:1787827552.102229:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827552.102232:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827552.102235:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827552.102243:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280651136/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827552.102266:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280650752:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827552.102300:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280650880:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827552.102322:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280651008:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827552.102349:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280651136:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787827553.126094:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827554.150055:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827555.174070:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787827555.174270:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787827555.174323:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827555.174357:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827555.174400:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827555.174407:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827555.174415:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827555.174424:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787827555.174471:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827552/real 1787827555] req@ffff8ab3f075cf00 x1874671280650752/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827555.174524:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280650752/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827555.174538:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827555.174545:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827555.174551:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280650752:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827555.174560:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827552/real 1787827555] req@ffff8ab3f075e580 x1874671280650880/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827555.174573:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280650880/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827555.174581:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827555.174585:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827555.174588:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280650880:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827555.174595:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827552/real 1787827555] req@ffff8ab3f075cb40 x1874671280651008/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827555.174614:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280651008/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827555.174624:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827555.174628:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827555.174632:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280651008:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827555.174638:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827552/real 1787827555] req@ffff8ab3f075da40 x1874671280651136/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827555.174650:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280651136/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827555.174678:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827555.174682:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827555.174686:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280651136:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827555.174693:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827552/real 1787827555] req@ffff8ab3f075f480 x1874671280650624/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827555.174708:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280650624/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827555.174716:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827555.174720:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827555.174723:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280650624:192.168.203.124@tcp:250:kworker.0 00010000:00010000:1.0:1787827556.198026:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827557.222008:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827557.222019:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827557.222023:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827557.222033:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2106 00000100:00080000:0.0:1787827557.222043:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827557.222051:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827557.222058:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827557.222087:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280651264/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827557.222113:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827557.222117:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827557.222120:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827557.222125:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2106 00000100:00080000:0.0:1787827557.222129:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827557.222135:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827557.222145:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280651264:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787827557.222147:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827557.222165:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280651392/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827557.222177:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827557.222181:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827557.222183:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827557.222188:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2106 00000100:00080000:0.0:1787827557.222193:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827557.222197:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787827557.222202:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787827557.222206:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827557.222220:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280651520/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827557.222231:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827557.222234:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827557.222237:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827557.222242:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2106 00000100:00080000:0.0:1787827557.222246:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827557.222254:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827557.222254:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827557/real 1787827557] req@ffff8ab3f075ed00 x1874671280651264/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827557.222262:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827557.222277:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280651648/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827557.222285:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280651264/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827557.222293:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827557.222301:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827557.222305:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827557.222312:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2106 00000100:00080000:1.0:1787827557.222312:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827557.222317:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827557.222322:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827557.222326:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787827557.222332:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827557.222339:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280651264:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827557.222342:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280651776/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827557.222373:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280651392:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827557.222402:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827557.222422:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827557/real 1787827557] req@ffff8ab3f075f480 x1874671280651392/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827557.222437:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280651392/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827557.222448:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827557.222452:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827557.222457:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280651392:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827557.222465:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280651520:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827557.222485:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827557.222496:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827557/real 1787827557] req@ffff8ab3f075da40 x1874671280651520/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827557.222515:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280651520/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827557.222523:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827557.222528:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827557.222531:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280651520:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827557.222536:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280651648:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827557.222557:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827557.222569:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827557/real 1787827557] req@ffff8ab3f075cb40 x1874671280651648/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827557.222580:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280651648/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827557.222589:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827557.222592:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827557.222595:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280651648:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827557.222601:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280651776:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827557.222619:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827557.222631:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827557/real 1787827557] req@ffff8ab3f075e580 x1874671280651776/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827557.222641:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280651776/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827557.222671:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827557.222675:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827557.222678:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280651776:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827558.246011:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827559.270115:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827560.294005:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827561.318022:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827562.342001:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827562.342016:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827562.342023:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827562.342038:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2111 00000100:00080000:3.0:1787827562.342055:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1787827562.342056:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827562.342083:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827562.342135:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280651904/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827562.342171:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827562.342175:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827562.342177:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827562.342180:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2111 00000100:00080000:3.0:1787827562.342184:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827562.342190:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827562.342204:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827562.342218:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280651904:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827562.342227:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280652032/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827562.342244:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827562.342247:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827562.342249:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827562.342255:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2111 00000100:00080000:3.0:1787827562.342258:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827562.342263:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827562.342270:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827562.342281:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280652160/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827562.342295:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827562.342298:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827562.342300:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827562.342303:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2111 00000100:00080000:3.0:1787827562.342306:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827562.342317:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827562.342323:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827562.342330:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827562.342349:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280652288/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827562.342375:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827562.342379:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827562.342381:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827562.342388:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2111 00000100:00080000:3.0:1787827562.342393:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827562.342398:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827562/real 1787827562] req@ffff8ab3d0a412c0 x1874671280651904/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827562.342402:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827562.342414:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827562.342458:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280652416/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827562.342459:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280651904/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827562.342507:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827562.342515:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827562.342523:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280651904:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827562.342556:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280652032:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827562.342578:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827562.342589:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827562/real 1787827562] req@ffff8ab3d0a41e00 x1874671280652032/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827562.342600:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280652032/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827562.342632:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827562.342636:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827562.342639:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280652032:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827562.342662:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280652160:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827562.342688:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827562.342706:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827562/real 1787827562] req@ffff8ab3d0a403c0 x1874671280652160/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827562.342731:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280652160/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827562.342737:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827562.342740:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827562.342743:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280652160:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827562.342748:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280652288:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827562.342763:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827562.342771:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827562/real 1787827562] req@ffff8ab3d0a421c0 x1874671280652288/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827562.342790:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280652288/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827562.342799:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827562.342801:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827562.342812:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280652288:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827562.342818:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280652416:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827562.342832:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827562.342840:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827562/real 1787827562] req@ffff8ab3d0a43c00 x1874671280652416/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827562.342860:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280652416/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827562.342868:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827562.342870:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827562.342872:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280652416:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827563.366128:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827564.390045:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827565.414115:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827566.438060:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827567.462049:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827567.462057:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1787827567.462057:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827567.462063:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827567.462073:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2117 00000100:00080000:1.0:1787827567.462081:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827567.462087:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827567.462116:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280652544/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827567.462138:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827567.462141:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827567.462142:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827567.462145:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2117 00000100:00080000:1.0:1787827567.462148:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827567.462152:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827567.462161:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827567.462167:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280652544:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827567.462172:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280652672/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827567.462181:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827567.462183:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827567.462184:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827567.462187:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2117 00000100:00080000:1.0:1787827567.462189:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827567.462192:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827567.462196:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827567.462204:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280652800/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827567.462210:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827567.462213:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827567.462215:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827567.462217:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2117 00000100:00080000:1.0:1787827567.462219:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827567.462223:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827567.462226:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787827567.462231:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827567.462235:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280652928/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827567.462243:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827567.462246:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827567.462249:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827567.462252:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2117 00000100:00080000:1.0:1787827567.462257:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827567.462261:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827567.462266:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827567.462273:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827567/real 1787827567] req@ffff8ab3d22fed00 x1874671280652544/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827567.462283:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280653056/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827567.462292:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280652544/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827567.462305:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827567.462309:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827567.462312:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280652544:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827567.462330:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280652672:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827567.462345:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827567.462354:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827567/real 1787827567] req@ffff8ab3d22fd2c0 x1874671280652672/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827567.462362:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280652672/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827567.462368:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827567.462370:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827567.462372:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280652672:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827567.462376:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280652800:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827567.462386:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827567.462392:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827567/real 1787827567] req@ffff8ab3d22fde00 x1874671280652800/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827567.462399:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280652800/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827567.462404:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827567.462406:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827567.462408:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280652800:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827567.462411:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280652928:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827567.462420:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827567.462425:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827567/real 1787827567] req@ffff8ab3d22fe1c0 x1874671280652928/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827567.462432:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280652928/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827567.462439:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827567.462441:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827567.462443:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280652928:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827567.462446:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280653056:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827567.462456:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827567.462461:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827567/real 1787827567] req@ffff8ab3d22fc780 x1874671280653056/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827567.462468:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280653056/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827567.462474:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827567.462476:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827567.462478:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280653056:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827568.486037:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827569.510087:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827570.534008:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827571.558035:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827572.582144:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827572.582161:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827572.582171:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827572.582178:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827572.582211:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2122 00000100:00080000:2.0:1787827572.582238:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827572.582247:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827572.582335:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280653184/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827572.582388:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827572.582391:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827572.582393:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827572.582396:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2122 00000100:00080000:2.0:1787827572.582400:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827572.582404:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827572.582431:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827572.582443:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280653184:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827572.582451:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280653312/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827572.582464:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827572.582467:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827572.582468:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827572.582473:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2122 00000100:00080000:2.0:1787827572.582476:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827572.582497:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827572.582503:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827572.582524:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280653440/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827572.582535:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827572.582561:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827572.582564:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827572.582570:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2122 00000100:00080000:2.0:1787827572.582576:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827572.582611:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827572.582619:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787827572.582627:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827572.582645:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280653568/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827572.582660:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827572.582684:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827572.582685:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827572.582692:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2122 00000100:00080000:2.0:1787827572.582698:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827572.582705:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827572.582712:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827572.582749:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280653696/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827572.582765:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827572/real 1787827572] req@ffff8ab3f0439e00 x1874671280653184/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827572.582791:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280653184/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827572.582825:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827572.582830:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827572.582836:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280653184:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827572.582873:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280653312:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827572.582921:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827572.582980:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827572/real 1787827572] req@ffff8ab3f043b840 x1874671280653312/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827572.582990:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280653312/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827572.582997:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827572.583000:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827572.583002:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280653312:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827572.583025:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280653440:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827572.583052:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827572.583066:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827572/real 1787827572] req@ffff8ab3f043b480 x1874671280653440/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827572.583080:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280653440/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827572.583107:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827572.583110:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827572.583114:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280653440:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827572.583121:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280653568:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827572.583142:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827572.583152:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827572/real 1787827572] req@ffff8ab3f043a940 x1874671280653568/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827572.583174:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280653568/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827572.583185:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827572.583188:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827572.583191:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280653568:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827572.583197:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280653696:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827572.583239:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827572.583248:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827572/real 1787827572] req@ffff8ab3f043a1c0 x1874671280653696/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827572.583258:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280653696/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827572.583268:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827572.583270:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827572.583272:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280653696:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827573.606120:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827574.630023:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827575.654092:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827576.678050:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827577.702036:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827577.702044:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827577.702048:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1787827577.702054:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827577.702057:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2127 00000100:00080000:3.0:1787827577.702068:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827577.702076:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827577.702114:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280653824/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827577.702142:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827577.702145:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827577.702146:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827577.702150:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2127 00000100:00080000:3.0:1787827577.702153:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827577.702157:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827577.702165:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827577.702176:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280653824:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827577.702178:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280653952/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827577.702190:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827577.702193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827577.702194:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827577.702197:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2127 00000100:00080000:3.0:1787827577.702199:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827577.702202:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827577.702204:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827577.702212:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280654080/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827577.702218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827577.702221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827577.702223:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827577.702226:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2127 00000100:00080000:3.0:1787827577.702229:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827577.702231:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827577.702234:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787827577.702236:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827577.702245:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280654208/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827577.702254:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827577.702256:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827577.702257:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827577.702260:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2127 00000100:00080000:3.0:1787827577.702262:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827577.702265:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827577.702268:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827577.702279:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827577/real 1787827577] req@ffff8ab3d0a43480 x1874671280653824/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827577.702279:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280654336/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827577.702311:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280653824/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827577.702321:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827577.702324:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827577.702327:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280653824:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827577.702344:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280653952:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827577.702358:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827577.702366:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827577/real 1787827577] req@ffff8ab3d0a43840 x1874671280653952/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827577.702374:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280653952/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827577.702390:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827577.702393:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827577.702395:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280653952:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827577.702398:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280654080:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827577.702407:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827577.702412:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827577/real 1787827577] req@ffff8ab3d0a430c0 x1874671280654080/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827577.702419:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280654080/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827577.702424:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827577.702426:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827577.702428:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280654080:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827577.702432:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280654208:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827577.702442:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827577.702454:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827577/real 1787827577] req@ffff8ab3d0a40000 x1874671280654208/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827577.702461:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280654208/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827577.702467:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827577.702468:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827577.702470:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280654208:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827577.702473:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280654336:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827577.702483:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827577.702488:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827577/real 1787827577] req@ffff8ab3d0a43c00 x1874671280654336/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827577.702495:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280654336/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827577.702501:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827577.702528:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827577.702530:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280654336:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827578.726022:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827579.750100:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827580.774027:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827581.798076:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827582.822030:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827582.822042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827582.822048:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1787827582.822050:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827582.822060:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2132 00000100:00080000:2.0:1787827582.822071:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827582.822081:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827582.822125:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280654464/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827582.822151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827582.822156:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827582.822158:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827582.822163:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2132 00000100:00080000:2.0:1787827582.822168:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827582.822174:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827582.822187:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827582.822195:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280654464:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827582.822202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280654592/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827582.822231:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827582.822234:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827582.822236:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827582.822240:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2132 00000100:00080000:2.0:1787827582.822244:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827582.822248:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827582.822253:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827582.822263:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280654720/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827582.822273:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827582.822273:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827582.822292:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827582.822294:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827582.822298:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2132 00000100:00080000:2.0:1787827582.822301:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827582.822305:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827582.822311:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827582.822323:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280654848/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827582.822334:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827582.822338:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827582.822340:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827582.822345:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2132 00000100:00080000:2.0:1787827582.822349:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827582.822354:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827582.822355:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827582/real 1787827582] req@ffff8ab3f0439a40 x1874671280654464/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827582.822359:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827582.822373:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280654976/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827582.822379:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280654464/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827582.822391:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827582.822396:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827582.822402:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280654464:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827582.822430:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280654592:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827582.822456:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827582.822499:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827582/real 1787827582] req@ffff8ab3f04392c0 x1874671280654592/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827582.822511:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280654592/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827582.822517:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827582.822520:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827582.822522:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280654592:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827582.822527:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280654720:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827582.822560:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827582.822569:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827582/real 1787827582] req@ffff8ab3f04383c0 x1874671280654720/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827582.822580:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280654720/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827582.822586:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827582.822588:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827582.822591:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280654720:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827582.822596:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280654848:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827582.822610:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827582.822617:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827582/real 1787827582] req@ffff8ab3f043bc00 x1874671280654848/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827582.822626:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280654848/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827582.822634:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827582.822637:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827582.822639:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280654848:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827582.822644:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280654976:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827582.822655:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827582.822662:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827582/real 1787827582] req@ffff8ab3f0439680 x1874671280654976/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827582.822671:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280654976/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827582.822685:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827582.822687:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827582.822690:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280654976:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827583.846065:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827584.870005:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827585.894091:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827586.918024:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827587.941993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827587.942005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827587.942011:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827587.942045:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2137 00000100:00080000:2.0:1787827587.942056:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1787827587.942060:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827587.942066:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827587.942108:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438000 x1874671280655104/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827587.942155:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827587.942158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827587.942160:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827587.942163:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2137 00000100:00080000:2.0:1787827587.942166:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827587.942170:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827587.942182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827587.942201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280655232/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827587.942218:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280655104:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827587.942218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827587.942246:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827587.942249:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827587.942256:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2137 00000100:00080000:2.0:1787827587.942261:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827587.942266:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827587.942272:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827587.942288:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280655360/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827587.942298:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827587.942302:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827587.942306:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827587.942307:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827587.942314:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2137 00000100:00080000:2.0:1787827587.942320:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827587.942325:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827587.942330:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827587.942344:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280655488/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827587.942353:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827587.942356:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827587.942358:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827587.942364:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2137 00000100:00080000:2.0:1787827587.942368:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827587.942372:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827587.942378:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827587/real 1787827587] req@ffff8ab3f0438000 x1874671280655104/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827587.942380:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827587.942399:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280655616/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827587.942408:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438000 x1874671280655104/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827587.942425:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827587.942429:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827587.942453:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280655104:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827587.942481:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280655232:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827587.942506:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827587.942519:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827587/real 1787827587] req@ffff8ab3f0438b40 x1874671280655232/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827587.942531:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280655232/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827587.942539:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827587.942543:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827587.942546:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280655232:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827587.942552:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280655360:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827587.942567:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827587.942575:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827587/real 1787827587] req@ffff8ab3f043b0c0 x1874671280655360/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827587.942586:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280655360/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827587.942593:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827587.942595:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827587.942598:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280655360:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827587.942602:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280655488:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827587.942616:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827587.942624:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827587/real 1787827587] req@ffff8ab3f043a580 x1874671280655488/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827587.942634:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280655488/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827587.942642:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827587.942645:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827587.942647:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280655488:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827587.942652:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280655616:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827587.942666:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827587.942690:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827587/real 1787827587] req@ffff8ab3f0438f00 x1874671280655616/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827587.942699:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280655616/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827587.942705:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827587.942707:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827587.942710:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280655616:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827588.966053:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827589.990083:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827591.014082:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827592.038020:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827593.062029:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827593.062041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827593.062046:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1787827593.062050:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827593.062060:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2142 00000100:00080000:2.0:1787827593.062074:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827593.062084:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827593.062139:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280655744/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827593.062171:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827593.062175:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827593.062177:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827593.062181:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2142 00000100:00080000:2.0:1787827593.062186:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827593.062190:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827593.062203:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827593.062228:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280655744:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827593.062228:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280655872/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827593.062260:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827593.062265:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827593.062268:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827593.062275:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2142 00000100:00080000:2.0:1787827593.062279:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827593.062283:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827593.062290:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827593.062325:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280656000/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827593.062336:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827593.062340:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827593.062342:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827593.062347:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2142 00000100:00080000:2.0:1787827593.062353:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827593.062355:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827593.062362:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827593.062370:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827593.062387:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280656128/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827593.062421:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827593.062425:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827593.062428:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827593.062451:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2142 00000100:00080000:2.0:1787827593.062455:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827593.062460:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827593.062469:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827593.062477:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827593/real 1787827593] req@ffff8ab3f043ad00 x1874671280655744/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827593.062488:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280656256/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827593.062508:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280655744/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827593.062525:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827593.062531:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827593.062536:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280655744:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827593.062570:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280655872:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827593.062600:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827593.062616:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827593/real 1787827593] req@ffff8ab3f0438780 x1874671280655872/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827593.062630:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280655872/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827593.062640:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827593.062644:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827593.062648:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280655872:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827593.062656:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280656000:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827593.062676:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827593.062685:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827593/real 1787827593] req@ffff8ab3f0438f00 x1874671280656000/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827593.062693:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280656000/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827593.062699:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827593.062701:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827593.062703:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280656000:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827593.062707:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280656128:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827593.062719:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827593.062725:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827593/real 1787827593] req@ffff8ab3f043a580 x1874671280656128/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827593.062733:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280656128/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827593.062741:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827593.062744:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827593.062746:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280656128:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827593.062750:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280656256:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827593.062760:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827593.062765:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827593/real 1787827593] req@ffff8ab3f043b0c0 x1874671280656256/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827593.062772:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280656256/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827593.062780:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827593.062781:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827593.062783:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280656256:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827594.086117:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827595.110060:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827596.134121:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827597.158022:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827598.182066:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827598.182094:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827598.182102:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827598.182106:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827598.182115:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2147 00000100:00080000:0.0:1787827598.182124:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827598.182131:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827598.182166:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280656384/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827598.182191:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827598.182194:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827598.182196:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827598.182199:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2147 00000100:00080000:0.0:1787827598.182204:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827598.182209:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827598.182223:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827598.182229:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280656384:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827598.182245:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280656512/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827598.182264:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827598.182268:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827598.182270:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827598.182274:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2147 00000100:00080000:0.0:1787827598.182280:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827598.182284:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827598.182289:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827598.182290:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827598.182307:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280656640/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827598.182317:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827598.182320:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827598.182322:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827598.182326:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2147 00000100:00080000:0.0:1787827598.182329:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827598.182333:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827598.182339:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827598/real 1787827598] req@ffff8ab3f075cf00 x1874671280656384/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827598.182341:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827598.182374:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280656768/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827598.182377:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280656384/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827598.182415:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827598.182418:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827598.182422:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827598.182430:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280656384:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827598.182431:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827598.182438:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827598.182446:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2147 00000100:00080000:0.0:1787827598.182451:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827598.182458:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280656512:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787827598.182460:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827598.182472:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787827598.182488:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827598.182492:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280656896/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827598.182505:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827598/real 1787827598] req@ffff8ab3f075f0c0 x1874671280656512/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827598.182520:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280656512/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827598.182528:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827598.182531:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827598.182534:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280656512:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827598.182539:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280656640:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827598.182552:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827598.182559:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827598/real 1787827598] req@ffff8ab3f075de00 x1874671280656640/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827598.182568:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280656640/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827598.182574:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827598.182577:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827598.182579:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280656640:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827598.182584:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280656768:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827598.182596:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827598.182619:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827598/real 1787827598] req@ffff8ab3f075f840 x1874671280656768/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827598.182631:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280656768/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827598.182642:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827598.182645:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827598.182648:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280656768:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827598.182687:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280656896:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827598.182708:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827598.182725:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827598/real 1787827598] req@ffff8ab3f075e940 x1874671280656896/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827598.182738:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280656896/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827598.182760:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827598.182764:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827598.182767:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280656896:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827599.206079:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827600.230159:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827601.254048:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827602.278120:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827603.301975:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827603.301987:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827603.301995:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827603.301999:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827603.302008:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2152 00000100:00080000:0.0:1787827603.302018:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827603.302035:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827603.302084:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280657024/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827603.302106:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827603.302109:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827603.302111:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827603.302114:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2152 00000100:00080000:0.0:1787827603.302117:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827603.302121:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827603.302130:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827603.302148:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280657152/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827603.302148:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280657024:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827603.302171:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827603.302175:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827603.302176:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827603.302180:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2152 00000100:00080000:0.0:1787827603.302183:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827603.302186:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827603.302191:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827603.302211:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280657280/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827603.302220:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827603.302223:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827603.302227:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827603.302228:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787827603.302233:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2152 00000100:00080000:0.0:1787827603.302237:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827603.302241:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827603.302246:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827603.302256:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280657408/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827603.302263:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827603.302267:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827603.302268:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827603.302271:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2152 00000100:00080000:0.0:1787827603.302276:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827603.302279:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827603/real 1787827603] req@ffff8ab3f075c000 x1874671280657024/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827603.302290:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827603.302296:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827603.302300:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280657024/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827603.302310:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280657536/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827603.302312:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827603.302321:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827603.302346:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280657024:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827603.302384:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280657152:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827603.302404:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827603.302416:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827603/real 1787827603] req@ffff8ab3f075e940 x1874671280657152/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827603.302433:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280657152/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827603.302440:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827603.302443:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827603.302446:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280657152:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827603.302450:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280657280:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827603.302462:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827603.302469:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827603/real 1787827603] req@ffff8ab3f075f840 x1874671280657280/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827603.302478:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280657280/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827603.302500:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827603.302503:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827603.302506:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280657280:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827603.302510:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280657408:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827603.302522:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827603.302529:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827603/real 1787827603] req@ffff8ab3f075de00 x1874671280657408/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827603.302538:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280657408/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827603.302546:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827603.302548:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827603.302550:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280657408:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827603.302554:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280657536:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827603.302565:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827603.302571:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827603/real 1787827603] req@ffff8ab3f075f0c0 x1874671280657536/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827603.302579:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280657536/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827603.302586:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827603.302588:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827603.302591:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280657536:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827604.326125:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827605.350023:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827606.374101:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827607.397988:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827608.421968:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827608.421977:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827608.421981:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827608.421989:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2158 00000100:00080000:0.0:1787827608.421998:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827608.422004:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787827608.422017:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827608.422040:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280657664/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827608.422059:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827608.422062:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827608.422064:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827608.422066:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2158 00000100:00080000:0.0:1787827608.422069:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827608.422071:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827608.422091:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827608.422102:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280657664:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827608.422107:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280657792/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827608.422122:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827608.422128:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827608.422131:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827608.422136:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2158 00000100:00080000:0.0:1787827608.422163:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827608.422167:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787827608.422171:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787827608.422175:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827608.422190:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280657920/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827608.422197:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827608.422200:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827608.422201:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827608.422205:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2158 00000100:00080000:0.0:1787827608.422208:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827608.422211:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827608.422215:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787827608.422216:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827608/real 1787827608] req@ffff8ab3f075cf00 x1874671280657664/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827608.422227:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280658048/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827608.422234:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280657664/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827608.422238:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827608.422247:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827608.422251:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827608.422254:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827608.422255:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2158 00000100:00080000:0.0:1787827608.422261:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827608.422262:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787827608.422266:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827608.422268:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280657664:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787827608.422273:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827608.422284:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280658176/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827608.422303:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280657792:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827608.422321:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827608.422331:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827608/real 1787827608] req@ffff8ab3f075e580 x1874671280657792/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827608.422340:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280657792/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827608.422345:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827608.422347:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827608.422349:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280657792:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827608.422353:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280657920:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827608.422364:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827608.422369:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827608/real 1787827608] req@ffff8ab3f075cb40 x1874671280657920/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827608.422376:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280657920/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827608.422381:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827608.422383:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827608.422385:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280657920:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827608.422388:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280658048:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827608.422398:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827608.422403:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827608/real 1787827608] req@ffff8ab3f075da40 x1874671280658048/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827608.422410:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280658048/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827608.422417:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827608.422418:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827608.422420:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280658048:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827608.422429:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280658176:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827608.422438:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827608.422444:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827608/real 1787827608] req@ffff8ab3f075f480 x1874671280658176/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827608.422467:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280658176/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827608.422473:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827608.422474:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827608.422476:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280658176:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827609.445985:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827610.470037:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827611.494017:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827612.518057:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827613.542041:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827613.542066:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827613.542071:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827613.542083:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2163 00000100:00080000:1.0:1787827613.542093:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827613.542112:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827613.542125:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827613.542164:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ffc00 x1874671280658304/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827613.542198:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827613.542204:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827613.542206:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827613.542210:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2163 00000100:00080000:1.0:1787827613.542215:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827613.542220:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827613.542232:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827613.542235:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ffc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280658304:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827613.542270:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280658432/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827613.542283:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827613.542303:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827613.542315:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827613.542319:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2163 00000100:00080000:1.0:1787827613.542337:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827613.542341:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827613.542346:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827613.542348:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827613.542361:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280658560/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827613.542371:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827613.542374:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827613.542376:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827613.542379:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2163 00000100:00080000:1.0:1787827613.542382:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827613.542386:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827613.542391:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827613.542402:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827613/real 1787827613] req@ffff8ab3d22ffc00 x1874671280658304/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827613.542404:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280658688/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827613.542445:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827613.542448:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827613.542450:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827613.542456:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2163 00000100:00100000:2.0:1787827613.542456:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ffc00 x1874671280658304/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827613.542459:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827613.542466:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827613.542475:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:2.0:1787827613.542482:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827613.542488:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280658816/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827613.542492:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827613.542503:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ffc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280658304:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827613.542532:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280658432:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827613.542549:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827613.542560:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827613/real 1787827613] req@ffff8ab3d22ff480 x1874671280658432/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827613.542570:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280658432/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827613.542577:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827613.542579:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827613.542582:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280658432:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827613.542587:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280658560:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827613.542599:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827613.542606:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827613/real 1787827613] req@ffff8ab3d22fda40 x1874671280658560/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827613.542615:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280658560/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827613.542621:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827613.542623:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827613.542625:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280658560:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827613.542629:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280658688:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827613.542642:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827613.542648:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827613/real 1787827613] req@ffff8ab3d22fcb40 x1874671280658688/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827613.542656:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280658688/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827613.542663:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827613.542666:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827613.542668:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280658688:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827613.542672:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280658816:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827613.542683:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827613.542690:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827613/real 1787827613] req@ffff8ab3d22fd680 x1874671280658816/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827613.542699:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280658816/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827613.542706:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827613.542708:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827613.542710:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280658816:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827614.566186:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827615.590125:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827616.614172:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827617.638121:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827618.662073:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827618.662087:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827618.662094:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827618.662109:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2168 00010000:00010000:3.0:1787827618.662111:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827618.662124:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827618.662137:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827618.662194:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280658944/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827618.662261:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827618.662266:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827618.662268:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827618.662273:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2168 00000100:00080000:0.0:1787827618.662277:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827618.662281:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827618.662294:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827618.662321:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280659072/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827618.662333:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827618.662338:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827618.662340:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827618.662344:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2168 00000100:00080000:0.0:1787827618.662351:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827618.662356:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827618.662371:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827618.662375:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280658944:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827618.662389:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280659200/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827618.662405:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827618.662409:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827618.662411:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827618.662416:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2168 00000100:00080000:0.0:1787827618.662421:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827618.662426:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827618.662432:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827618.662444:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280659328/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827618.662453:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827618.662457:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827618.662459:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827618.662471:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2168 00000100:00080000:0.0:1787827618.662474:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827618.662478:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827618.662483:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827618.662493:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280659456/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827618.662556:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280659072:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827618.662624:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280659200:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827618.662691:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280659328:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827618.662729:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280659456:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827619.686370:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827620.710276:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827621.734263:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827621.734327:0:10942:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787827621.734378:0:10942:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827621.734399:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827621.734456:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827621.734470:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827621.734484:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827621.734492:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787827621.734516:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827618/real 1787827621] req@ffff8ab3f075da40 x1874671280659200/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827621.734553:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280659200/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827621.734568:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827621.734574:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827621.734579:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280659200:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827621.734588:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827618/real 1787827621] req@ffff8ab3f075cb40 x1874671280659328/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827621.734601:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280659328/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827621.734611:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827621.734614:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827621.734617:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280659328:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827621.734624:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827618/real 1787827621] req@ffff8ab3f075e580 x1874671280659456/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827621.734643:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280659456/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827621.734652:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827621.734655:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827621.734659:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280659456:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827621.734665:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827618/real 1787827621] req@ffff8ab3f075ed00 x1874671280658944/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827621.734677:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280658944/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827621.734685:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827621.734689:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827621.734692:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280658944:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827621.734699:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827618/real 1787827621] req@ffff8ab3f075f480 x1874671280659072/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827621.734710:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280659072/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827621.734718:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827621.734721:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827621.734724:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280659072:192.168.203.124@tcp:38:kworker.0 00010000:00010000:3.0:1787827622.758210:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827623.782007:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827623.782021:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827623.782025:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827623.782035:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2173 00000100:00080000:2.0:1787827623.782044:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827623.782052:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827623.782087:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280659584/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827623.782117:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827623.782121:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827623.782124:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827623.782128:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2173 00000100:00080000:2.0:1787827623.782131:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827623.782135:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787827623.782142:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:2.0:1787827623.782146:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827623.782150:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280659584:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827623.782166:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438000 x1874671280659712/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827623.782176:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827623.782180:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827623.782196:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827623.782200:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2173 00000100:00080000:2.0:1787827623.782218:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827623.782221:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827623.782225:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827623.782230:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827623.782240:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280659840/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827623.782250:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827623.782253:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827623.782255:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827623.782258:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2173 00000100:00080000:2.0:1787827623.782263:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827623.782266:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827623.782270:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827623.782283:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280659968/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827623.782292:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827623/real 1787827623] req@ffff8ab3f0438b40 x1874671280659584/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827623.782295:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827623.782307:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827623.782313:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827623.782321:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280659584/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827623.782322:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2173 00000100:00080000:2.0:1787827623.782332:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827623.782339:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827623.782341:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787827623.782346:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000008:00100000:2.0:1787827623.782352:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827623.782353:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280659584:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827623.782377:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280660096/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827623.782387:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280659712:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827623.782433:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827623.782452:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827623/real 1787827623] req@ffff8ab3f0438000 x1874671280659712/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827623.782468:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438000 x1874671280659712/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827623.782479:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827623.782493:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827623.782497:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280659712:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827623.782505:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280659840:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827623.782540:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827623.782555:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827623/real 1787827623] req@ffff8ab3f0439680 x1874671280659840/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827623.782579:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280659840/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827623.782589:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827623.782594:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827623.782598:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280659840:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827623.782606:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280659968:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827623.782630:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827623.782644:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827623/real 1787827623] req@ffff8ab3f043bc00 x1874671280659968/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827623.782658:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280659968/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827623.782669:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827623.782673:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827623.782676:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280659968:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827623.782694:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280660096:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827623.782718:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827623.782741:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827623/real 1787827623] req@ffff8ab3f04383c0 x1874671280660096/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827623.782757:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280660096/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827623.782777:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827623.782782:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827623.782787:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280660096:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827624.806265:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827625.830104:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827626.854078:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827627.878101:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827628.901962:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827628.901980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827628.901984:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827628.901992:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2178 00000100:00080000:2.0:1787827628.902001:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827628.902007:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787827628.902019:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827628.902041:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280660224/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827628.902062:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827628.902073:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827628.902074:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827628.902078:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2178 00000100:00080000:2.0:1787827628.902080:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827628.902083:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827628.902092:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827628.902095:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280660224:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827628.902102:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280660352/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827628.902110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827628.902113:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827628.902114:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827628.902117:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2178 00000100:00080000:2.0:1787827628.902119:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827628.902121:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827628.902125:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827628.902133:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280660480/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827628.902139:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827628.902142:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827628.902143:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827628.902145:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2178 00000100:00080000:2.0:1787827628.902161:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827628.902165:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787827628.902166:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787827628.902172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827628.902182:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280660608/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827628.902199:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827628.902201:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827628.902202:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827628.902205:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2178 00000100:00080000:2.0:1787827628.902208:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827628.902210:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827628.902214:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827628.902221:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827628/real 1787827628] req@ffff8ab3f04392c0 x1874671280660224/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827628.902223:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280660736/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827628.902255:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280660224/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827628.902265:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827628.902268:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827628.902271:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280660224:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827628.902290:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280660352:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827628.902311:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827628.902325:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827628/real 1787827628] req@ffff8ab3f0439a40 x1874671280660352/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827628.902338:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280660352/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827628.902355:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827628.902358:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827628.902361:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280660352:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827628.902366:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280660480:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827628.902381:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827628.902389:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827628/real 1787827628] req@ffff8ab3f043a1c0 x1874671280660480/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827628.902396:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280660480/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827628.902402:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827628.902404:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827628.902406:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280660480:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827628.902410:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280660608:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827628.902420:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827628.902425:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827628/real 1787827628] req@ffff8ab3f043a940 x1874671280660608/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827628.902432:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280660608/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827628.902438:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827628.902440:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827628.902442:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280660608:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827628.902445:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280660736:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827628.902455:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827628.902460:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827628/real 1787827628] req@ffff8ab3f043b480 x1874671280660736/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827628.902467:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280660736/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827628.902473:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827628.902475:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827628.902477:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280660736:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827629.926072:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827630.950113:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827631.974247:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827632.998186:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827634.022015:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827634.022030:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827634.022037:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827634.022053:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2183 00000100:00080000:2.0:1787827634.022069:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827634.022081:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787827634.022134:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827634.022171:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280660864/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827634.022241:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827634.022248:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827634.022252:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827634.022262:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2183 00000100:00080000:2.0:1787827634.022270:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827634.022280:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827634.022302:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827634.022303:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280660864:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827634.022338:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280660992/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827634.022358:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827634.022363:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827634.022366:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827634.022372:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2183 00000100:00080000:2.0:1787827634.022393:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827634.022399:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787827634.022409:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787827634.022410:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827634.022438:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280661120/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827634.022452:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827634.022458:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827634.022461:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827634.022467:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2183 00000100:00080000:2.0:1787827634.022474:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827634.022480:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827634.022488:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787827634.022510:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827634/real 1787827634] req@ffff8ab3f043b840 x1874671280660864/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827634.022526:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280661248/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827634.022571:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827634.022579:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827634.022581:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827634.022588:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2183 00000100:00080000:2.0:1787827634.022592:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827634.022619:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280660864/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827634.022620:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827634.022637:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787827634.022661:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827634.022672:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280661376/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827634.022676:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827634.022712:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280660864:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827634.022754:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280660992:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827634.022804:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827634.022824:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827634/real 1787827634] req@ffff8ab3f0439e00 x1874671280660992/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827634.022842:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280660992/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827634.022863:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827634.022867:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827634.022872:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280660992:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827634.022880:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280661120:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827634.022970:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827634.022993:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827634/real 1787827634] req@ffff8ab3f043b480 x1874671280661120/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827634.023017:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280661120/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827634.023030:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827634.023034:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827634.023038:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280661120:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827634.023046:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280661248:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827634.023074:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827634.023087:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827634/real 1787827634] req@ffff8ab3f043a940 x1874671280661248/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827634.023100:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280661248/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827634.023144:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827634.023150:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827634.023156:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280661248:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827634.023197:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280661376:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827634.023234:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827634.023257:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827634/real 1787827634] req@ffff8ab3f043a1c0 x1874671280661376/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827634.023282:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280661376/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827634.023303:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827634.023310:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827634.023317:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280661376:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827635.046229:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827636.070081:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827637.094229:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827638.118170:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827639.142034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827639.142042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827639.142045:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827639.142055:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2188 00000100:00080000:3.0:1787827639.142063:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827639.142072:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827639.142094:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827639.142128:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280661504/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827639.142150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827639.142153:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827639.142154:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827639.142157:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2188 00000100:00080000:3.0:1787827639.142159:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827639.142161:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827639.142170:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827639.142179:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280661632/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827639.142188:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827639.142190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827639.142191:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827639.142198:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2188 00000100:00100000:1.0:1787827639.142200:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280661504:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827639.142202:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827639.142213:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827639.142218:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827639.142225:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280661760/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827639.142231:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827639.142233:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827639.142235:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827639.142238:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2188 00000100:00080000:3.0:1787827639.142241:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827639.142245:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827639.142249:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827639.142266:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280661888/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827639.142274:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827639.142277:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827639.142279:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827639.142281:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2188 00000100:00080000:3.0:1787827639.142292:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827639.142295:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827639.142298:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787827639.142304:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827639.142309:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280662016/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827639.142362:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827639/real 1787827639] req@ffff8ab3d0a421c0 x1874671280661504/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.142378:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280661504/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.142388:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.142392:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827639.142395:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280661504:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827639.142418:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280661632:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827639.142433:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827639.142443:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827639/real 1787827639] req@ffff8ab3d0a403c0 x1874671280661632/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.142451:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280661632/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.142457:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.142460:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827639.142462:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280661632:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827639.142466:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280661760:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827639.142476:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827639.142481:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827639/real 1787827639] req@ffff8ab3d0a41e00 x1874671280661760/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.142489:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280661760/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.142494:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.142496:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827639.142497:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280661760:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827639.142501:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280661888:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827639.142511:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827639.142517:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827639/real 1787827639] req@ffff8ab3d0a412c0 x1874671280661888/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.142524:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280661888/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.142532:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.142534:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827639.142536:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280661888:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827639.142540:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280662016:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827639.142550:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827639.142555:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827639/real 1787827639] req@ffff8ab3d0a43c00 x1874671280662016/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.142562:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280662016/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.142569:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.142571:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827639.142573:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280662016:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827640.166017:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827641.190121:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827642.214042:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827643.238093:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827644.262002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827644.262011:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827644.262015:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827644.262025:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2193 00010000:00010000:3.0:1787827644.262030:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827644.262034:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827644.262057:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827644.262092:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280662144/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827644.262123:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827644.262128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827644.262131:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827644.262136:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2193 00000100:00080000:2.0:1787827644.262141:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827644.262147:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827644.262159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827644.262173:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280662144:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827644.262178:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280662272/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827644.262194:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827644.262198:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827644.262201:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827644.262207:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2193 00000100:00080000:2.0:1787827644.262225:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827644.262232:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787827644.262237:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787827644.262242:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827644.262261:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280662400/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827644.262273:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827644.262278:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827644.262280:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827644.262286:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2193 00000100:00100000:1.0:1787827644.262291:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827644/real 1787827644] req@ffff8ab3f0439a40 x1874671280662144/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827644.262293:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827644.262304:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827644.262314:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280662144/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827644.262314:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787827644.262329:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827644.262336:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827644.262338:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280662528/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827644.262341:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280662144:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827644.262358:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827644.262363:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827644.262365:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827644.262368:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280662272:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787827644.262374:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2193 00000100:00080000:2.0:1787827644.262380:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827644.262386:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787827644.262392:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787827644.262395:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827644.262405:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827644/real 1787827644] req@ffff8ab3f04392c0 x1874671280662272/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827644.262415:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280662656/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827644.262420:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280662272/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827644.262446:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827644.262450:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827644.262454:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280662272:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827644.262460:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280662400:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827644.262473:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827644.262481:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827644/real 1787827644] req@ffff8ab3f04383c0 x1874671280662400/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827644.262489:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280662400/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827644.262495:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827644.262497:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827644.262499:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280662400:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827644.262503:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280662528:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827644.262514:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827644.262521:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827644/real 1787827644] req@ffff8ab3f043bc00 x1874671280662528/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827644.262529:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280662528/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827644.262536:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827644.262538:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827644.262540:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280662528:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827644.262550:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280662656:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827644.262561:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827644.262568:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827644/real 1787827644] req@ffff8ab3f0439680 x1874671280662656/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827644.262576:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280662656/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827644.262583:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827644.262585:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827644.262587:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280662656:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827645.286063:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827646.310074:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827647.334045:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827648.358150:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827649.381994:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827649.382019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827649.382024:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827649.382035:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2199 00010000:00010000:2.0:1787827649.382043:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827649.382046:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827649.382055:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827649.382092:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280662784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827649.382119:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827649.382123:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827649.382125:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827649.382129:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2199 00000100:00080000:3.0:1787827649.382133:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827649.382139:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827649.382150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827649.382156:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280662784:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827649.382167:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280662912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827649.382177:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827649.382180:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827649.382182:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827649.382185:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2199 00000100:00080000:3.0:1787827649.382206:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827649.382212:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827649.382213:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827649.382220:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827649.382234:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280663040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827649.382245:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827649.382249:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827649.382251:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827649.382256:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827649/real 1787827649] req@ffff8ab3d0a40000 x1874671280662784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827649.382261:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2199 00000100:00080000:3.0:1787827649.382271:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827649.382278:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280662784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827649.382279:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827649.382288:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787827649.382292:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827649.382297:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827649.382301:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280662784:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827649.382304:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280663168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827649.382314:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827649.382316:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827649.382317:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827649.382321:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2199 00000100:00100000:1.0:1787827649.382324:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280662912:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787827649.382326:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827649.382333:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827649.382337:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827649.382347:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280663296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827649.382354:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827649.382368:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827649/real 1787827649] req@ffff8ab3d0a430c0 x1874671280662912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827649.382377:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280662912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827649.382383:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827649.382385:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827649.382387:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280662912:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827649.382391:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280663040:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787827649.382410:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827649.382416:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827649/real 1787827649] req@ffff8ab3d0a43840 x1874671280663040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827649.382423:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280663040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827649.382428:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827649.382430:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827649.382432:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280663040:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787827649.382436:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280663168:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827649.382446:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827649.382451:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827649/real 1787827649] req@ffff8ab3d0a43480 x1874671280663168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827649.382458:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280663168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827649.382464:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827649.382466:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827649.382468:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280663168:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787827649.382477:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280663296:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787827649.382486:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827649.382492:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827649/real 1787827649] req@ffff8ab3d0a40000 x1874671280663296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827649.382499:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280663296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827649.382512:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827649.382514:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787827649.382516:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280663296:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827650.406032:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827651.430190:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827652.454048:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827653.478121:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827654.502065:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1787827654.502067:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827654.502090:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827654.502098:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827654.502111:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2204 00000100:00080000:1.0:1787827654.502122:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827654.502129:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827654.502174:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280663424/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827654.502215:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827654.502218:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827654.502220:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827654.502223:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2204 00000100:00080000:1.0:1787827654.502225:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827654.502228:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827654.502237:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827654.502247:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280663552/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827654.502255:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827654.502258:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827654.502260:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827654.502263:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2204 00000100:00080000:1.0:1787827654.502265:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827654.502267:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827654.502283:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827654.502295:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280663680/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827654.502316:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827654.502326:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787827654.502327:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280663424:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787827654.502329:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827654.502351:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2204 00000100:00080000:1.0:1787827654.502355:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827654.502359:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827654.502364:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787827654.502388:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280663808/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827654.502397:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827654.502400:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827654.502401:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827654.502404:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2204 00000100:00080000:1.0:1787827654.502423:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827654.502426:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827654.502429:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:2.0:1787827654.502435:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827654.502440:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280663936/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827654.502507:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827654/real 1787827654] req@ffff8ab3d22fc000 x1874671280663424/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827654.502529:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280663424/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827654.502543:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827654.502550:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827654.502555:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280663424:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827654.502565:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280663552:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827654.502590:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827654.502605:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827654/real 1787827654] req@ffff8ab3d22fd680 x1874671280663552/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827654.502618:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280663552/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827654.502627:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827654.502631:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827654.502635:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280663552:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827654.502663:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280663680:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827654.502684:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827654.502698:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827654/real 1787827654] req@ffff8ab3d22fcb40 x1874671280663680/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827654.502710:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280663680/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827654.502719:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827654.502722:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827654.502726:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280663680:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827654.502733:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280663808:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827654.502753:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827654.502765:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827654/real 1787827654] req@ffff8ab3d22fda40 x1874671280663808/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827654.502790:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280663808/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827654.502805:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827654.502809:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827654.502812:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280663808:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827654.502819:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280663936:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827654.502840:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827654.502853:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827654/real 1787827654] req@ffff8ab3d22ff480 x1874671280663936/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827654.502866:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280663936/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827654.502877:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827654.502881:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827654.502927:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280663936:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827655.526101:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827656.550127:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827657.574057:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827658.598054:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827659.621982:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827659.621990:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827659.621994:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827659.622002:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2209 00000100:00080000:1.0:1787827659.622010:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827659.622017:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787827659.622050:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ffc00 x1874671280664064/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827659.622076:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827659.622079:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827659.622080:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827659.622083:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2209 00000100:00080000:1.0:1787827659.622086:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827659.622090:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827659.622099:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827659.622115:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280664192/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827659.622115:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ffc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280664064:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787827659.622130:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827659.622132:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827659.622133:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827659.622136:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2209 00000100:00080000:1.0:1787827659.622139:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827659.622141:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827659.622145:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827659.622154:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280664320/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827659.622161:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827659.622163:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827659.622165:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827659.622168:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2209 00000100:00080000:1.0:1787827659.622171:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827659.622173:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:3.0:1787827659.622177:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:1.0:1787827659.622179:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787827659.622189:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280664448/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827659.622197:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827659.622199:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827659.622201:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827659.622205:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2209 00000100:00080000:1.0:1787827659.622207:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827659.622209:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827659.622212:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827659.622222:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280664576/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827659.622233:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827659/real 1787827659] req@ffff8ab3d22ffc00 x1874671280664064/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827659.622255:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ffc00 x1874671280664064/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827659.622263:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827659.622267:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827659.622270:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ffc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280664064:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827659.622292:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280664192:192.168.203.124@tcp:38:kworker.0 00010000:00010000:2.0:1787827659.622299:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1787827659.622310:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827659.622320:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827659/real 1787827659] req@ffff8ab3d22fc780 x1874671280664192/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827659.622328:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280664192/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827659.622333:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827659.622335:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827659.622337:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280664192:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827659.622341:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280664320:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827659.622351:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827659.622357:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827659/real 1787827659] req@ffff8ab3d22fe1c0 x1874671280664320/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827659.622364:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280664320/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827659.622373:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827659.622375:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827659.622377:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280664320:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827659.622380:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280664448:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827659.622390:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827659.622395:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827659/real 1787827659] req@ffff8ab3d22fde00 x1874671280664448/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827659.622402:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280664448/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827659.622408:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827659.622410:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827659.622411:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280664448:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827659.622415:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280664576:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827659.622424:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827659.622429:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827659/real 1787827659] req@ffff8ab3d22fd2c0 x1874671280664576/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827659.622436:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280664576/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827659.622442:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827659.622443:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827659.622446:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280664576:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827660.646081:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827661.670045:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827662.694060:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827663.718075:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827664.742049:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827664.742059:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827664.742064:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827664.742075:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2214 00000100:00080000:0.0:1787827664.742085:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827664.742094:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787827664.742112:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827664.742132:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280664704/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827664.742160:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827664.742165:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827664.742167:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827664.742171:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2214 00000100:00080000:0.0:1787827664.742176:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827664.742181:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827664.742192:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827664.742196:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280664704:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827664.742208:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280664832/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827664.742222:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827664.742226:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827664.742229:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827664.742234:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2214 00000100:00080000:0.0:1787827664.742247:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827664.742253:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787827664.742254:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827664.742263:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827664.742277:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280664960/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827664.742288:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827664.742292:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787827664.742295:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827664/real 1787827664] req@ffff8ab3f075cf00 x1874671280664704/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827664.742296:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827664.742307:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2214 00000100:00080000:0.0:1787827664.742313:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827664.742316:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280664704/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827664.742323:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827664.742329:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:3.0:1787827664.742338:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827664.742344:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280665088/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827664.742345:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827664.742352:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280664704:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827664.742368:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827664.742372:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827664.742374:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827664.742379:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280664832:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787827664.742383:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2214 00000100:00080000:0.0:1787827664.742389:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827664.742394:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827664.742400:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:3.0:1787827664.742406:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827664.742417:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280665216/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827664.742420:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827664/real 1787827664] req@ffff8ab3f075f0c0 x1874671280664832/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827664.742454:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280664832/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827664.742461:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827664.742464:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827664.742466:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280664832:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827664.742474:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280664960:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827664.742486:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827664.742496:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827664/real 1787827664] req@ffff8ab3f075de00 x1874671280664960/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827664.742510:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280664960/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827664.742518:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827664.742522:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827664.742525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280664960:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827664.742531:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280665088:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827664.742547:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827664.742561:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827664/real 1787827664] req@ffff8ab3f075f840 x1874671280665088/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827664.742568:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280665088/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827664.742575:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827664.742577:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827664.742579:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280665088:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827664.742591:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280665216:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827664.742602:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827664.742608:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827664/real 1787827664] req@ffff8ab3f075e940 x1874671280665216/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827664.742616:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280665216/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827664.742622:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827664.742624:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827664.742626:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280665216:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827665.766055:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827666.790101:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827667.814066:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827668.838059:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827669.861973:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827669.861984:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827669.861989:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827669.862002:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2219 00000100:00080000:0.0:1787827669.862012:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827669.862021:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827669.862065:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827669.862068:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280665344/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827669.862095:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827669.862100:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827669.862102:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827669.862106:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2219 00000100:00080000:0.0:1787827669.862110:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827669.862115:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827669.862127:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827669.862144:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280665472/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827669.862153:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280665344:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827669.862155:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827669.862161:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827669.862162:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827669.862167:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2219 00000100:00080000:0.0:1787827669.862170:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827669.862174:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827669.862179:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827669.862189:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280665600/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827669.862197:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827669.862200:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827669.862202:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827669.862206:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2219 00000100:00080000:0.0:1787827669.862218:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827669.862223:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827669.862228:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:3.0:1787827669.862228:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827669.862238:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280665728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827669.862246:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827669.862249:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827669.862251:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827669.862255:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2219 00000100:00080000:0.0:1787827669.862258:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827669.862263:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827669.862277:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827669.862285:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827669/real 1787827669] req@ffff8ab3f075c000 x1874671280665344/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827669.862291:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280665856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827669.862304:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280665344/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827669.862315:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827669.862320:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827669.862324:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280665344:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827669.862356:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280665472:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827669.862376:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827669.862386:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827669/real 1787827669] req@ffff8ab3f075e940 x1874671280665472/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827669.862394:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280665472/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827669.862401:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827669.862403:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827669.862405:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280665472:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827669.862409:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280665600:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827669.862419:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827669.862424:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827669/real 1787827669] req@ffff8ab3f075f840 x1874671280665600/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827669.862431:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280665600/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827669.862437:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827669.862439:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827669.862440:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280665600:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827669.862444:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280665728:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827669.862455:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827669.862460:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827669/real 1787827669] req@ffff8ab3f075de00 x1874671280665728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827669.862468:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280665728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827669.862474:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827669.862476:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827669.862479:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280665728:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827669.862483:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280665856:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827669.862493:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827669.862498:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827669/real 1787827669] req@ffff8ab3f075f0c0 x1874671280665856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827669.862505:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280665856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827669.862512:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827669.862513:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827669.862515:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280665856:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827670.886053:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827671.910029:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827672.934040:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827673.958020:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827674.982007:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827674.982015:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827674.982019:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827674.982030:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2224 00010000:00010000:3.0:1787827674.982038:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827674.982039:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827674.982047:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827674.982079:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438000 x1874671280665984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827674.982098:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827674.982101:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827674.982102:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827674.982104:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2224 00000100:00080000:2.0:1787827674.982107:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827674.982110:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827674.982119:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827674.982128:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280665984:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827674.982129:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280666112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827674.982143:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827674.982145:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827674.982147:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827674.982150:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2224 00000100:00080000:2.0:1787827674.982152:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827674.982154:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827674.982159:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827674.982167:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280666240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827674.982173:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827674.982176:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827674.982177:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827674.982180:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2224 00000100:00080000:2.0:1787827674.982182:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827674.982185:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827674.982188:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:3.0:1787827674.982190:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827674.982200:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280666368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827674.982208:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827674.982210:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827674.982212:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827674.982214:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2224 00000100:00080000:2.0:1787827674.982216:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827674.982219:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827674.982222:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827674.982228:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280666496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827674.982247:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827674/real 1787827674] req@ffff8ab3f0438000 x1874671280665984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827674.982262:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438000 x1874671280665984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827674.982270:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827674.982273:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827674.982276:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280665984:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827674.982296:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280666112:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827674.982311:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827674.982320:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827674/real 1787827674] req@ffff8ab3f0438b40 x1874671280666112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827674.982328:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280666112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827674.982333:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827674.982335:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827674.982338:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280666112:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827674.982342:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280666240:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827674.982351:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827674.982356:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827674/real 1787827674] req@ffff8ab3f043b0c0 x1874671280666240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827674.982363:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280666240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827674.982368:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827674.982370:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827674.982371:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280666240:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827674.982375:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280666368:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827674.982384:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827674.982391:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827674/real 1787827674] req@ffff8ab3f043a580 x1874671280666368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827674.982403:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280666368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827674.982413:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827674.982417:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827674.982420:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280666368:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827674.982426:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280666496:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827674.982446:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827674.982458:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827674/real 1787827674] req@ffff8ab3f0438f00 x1874671280666496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827674.982471:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280666496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827674.982481:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827674.982485:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827674.982488:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280666496:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827676.006066:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827677.030070:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827678.054080:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827679.078018:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827680.102005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827680.102016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1787827680.102021:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827680.102022:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827680.102034:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2229 00000100:00080000:3.0:1787827680.102045:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827680.102055:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827680.102094:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280666624/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827680.102117:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827680.102120:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827680.102121:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827680.102124:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2229 00000100:00080000:3.0:1787827680.102127:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827680.102130:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827680.102139:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827680.102148:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280666752/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827680.102157:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827680.102161:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280666624:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827680.102161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827680.102165:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827680.102173:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2229 00000100:00080000:3.0:1787827680.102175:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827680.102178:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827680.102182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827680.102191:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280666880/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827680.102200:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827680.102205:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827680.102207:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827680.102212:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2229 00000100:00080000:3.0:1787827680.102216:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827680.102223:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827680.102224:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827680.102234:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827680.102250:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280667008/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827680.102260:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827680.102263:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827680.102267:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827680.102268:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827680/real 1787827680] req@ffff8ab3d0a43c00 x1874671280666624/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827680.102273:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2229 00000100:00080000:3.0:1787827680.102279:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827680.102286:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827680.102291:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827680.102295:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280666624/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827680.102304:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280667136/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827680.102306:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827680.102316:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827680.102321:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280666624:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827680.102342:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280666752:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827680.102359:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827680.102368:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827680/real 1787827680] req@ffff8ab3d0a412c0 x1874671280666752/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827680.102377:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280666752/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827680.102383:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827680.102385:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827680.102388:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280666752:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827680.102392:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280666880:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827680.102401:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827680.102407:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827680/real 1787827680] req@ffff8ab3d0a41e00 x1874671280666880/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827680.102416:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280666880/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827680.102421:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827680.102423:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827680.102425:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280666880:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827680.102428:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280667008:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827680.102440:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827680.102446:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827680/real 1787827680] req@ffff8ab3d0a403c0 x1874671280667008/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827680.102454:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280667008/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827680.102461:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827680.102463:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827680.102465:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280667008:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827680.102471:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280667136:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827680.102491:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827680.102503:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827680/real 1787827680] req@ffff8ab3d0a421c0 x1874671280667136/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827680.102517:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280667136/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827680.102534:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827680.102537:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827680.102540:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280667136:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827681.125996:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827682.150149:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827683.174027:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827684.198056:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827685.221984:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827685.221993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827685.221996:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827685.222006:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2234 00000100:00080000:2.0:1787827685.222014:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827685.222021:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787827685.222044:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827685.222055:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280667264/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827685.222081:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827685.222083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827685.222085:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827685.222088:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2234 00000100:00080000:2.0:1787827685.222091:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827685.222094:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827685.222102:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827685.222108:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280667264:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827685.222114:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280667392/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827685.222122:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827685.222125:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827685.222126:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827685.222129:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2234 00000100:00080000:2.0:1787827685.222131:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827685.222134:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827685.222137:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827685.222146:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280667520/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827685.222153:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827685.222155:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827685.222157:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827685.222161:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2234 00000100:00080000:2.0:1787827685.222165:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827685.222170:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827685.222176:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827685.222189:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280667648/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827685.222199:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827685.222205:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280667392:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787827685.222216:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827685.222219:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827685.222225:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2234 00000100:00100000:3.0:1787827685.222230:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280667520:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787827685.222231:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827685.222241:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827685.222248:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827685.222251:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280667648:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827685.222267:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280667776/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827685.222304:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280667776:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827686.246040:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827687.270047:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827688.294076:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787827688.294273:0:10944:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787827688.294324:0:10944:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827688.294337:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827688.294380:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827688.294393:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827688.294404:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827688.294414:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:3.0:1787827688.294425:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827685/real 1787827688] req@ffff8ab3f043ad00 x1874671280667392/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827688.294454:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280667392/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827688.294469:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827688.294474:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827688.294479:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280667392:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827688.294487:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827685/real 1787827688] req@ffff8ab3f0438f00 x1874671280667520/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827688.294501:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280667520/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827688.294510:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827688.294514:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827688.294518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280667520:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827688.294525:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827685/real 1787827688] req@ffff8ab3f043a580 x1874671280667648/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827688.294538:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280667648/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827688.294549:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827688.294553:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827688.294557:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280667648:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827688.294563:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827685/real 1787827688] req@ffff8ab3f0438780 x1874671280667264/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827688.294575:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280667264/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827688.294584:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827688.294587:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827688.294597:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280667264:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827688.294625:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827685/real 1787827688] req@ffff8ab3f043b0c0 x1874671280667776/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827688.294639:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280667776/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827688.294650:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827688.294653:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827688.294658:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280667776:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827689.318039:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827690.341969:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827690.341976:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827690.341979:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827690.341988:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2239 00000100:00080000:2.0:1787827690.341996:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827690.342003:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827690.342034:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280667904/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787827690.342038:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827690.342053:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827690.342056:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827690.342059:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827690.342066:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2239 00000100:00080000:2.0:1787827690.342069:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827690.342074:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827690.342082:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827690.342082:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280667904:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827690.342093:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438000 x1874671280668032/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827690.342102:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827690.342104:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827690.342105:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827690.342108:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2239 00000100:00080000:2.0:1787827690.342111:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827690.342121:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827690.342125:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827690.342135:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280668160/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827690.342142:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827690.342148:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827690.342151:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827690.342152:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827690.342156:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2239 00000100:00080000:2.0:1787827690.342160:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827690.342163:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827690.342171:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827690.342179:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280668288/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827690.342186:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827690.342191:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787827690.342191:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827690/real 1787827690] req@ffff8ab3f0438b40 x1874671280667904/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827690.342193:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827690.342200:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2239 00000100:00080000:2.0:1787827690.342204:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827690.342210:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827690.342215:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827690.342226:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280668416/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827690.342229:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280667904/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827690.342247:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827690.342250:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827690.342254:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280667904:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827690.342271:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280668032:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827690.342286:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827690.342294:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827690/real 1787827690] req@ffff8ab3f0438000 x1874671280668032/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827690.342307:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438000 x1874671280668032/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827690.342313:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827690.342315:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827690.342318:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280668032:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827690.342321:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280668160:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827690.342331:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827690.342337:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827690/real 1787827690] req@ffff8ab3f0439680 x1874671280668160/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827690.342344:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280668160/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827690.342353:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827690.342356:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827690.342358:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280668160:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827690.342361:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280668288:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827690.342371:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827690.342378:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827690/real 1787827690] req@ffff8ab3f043bc00 x1874671280668288/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827690.342390:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280668288/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827690.342400:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827690.342403:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827690.342406:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280668288:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827690.342412:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280668416:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827690.342431:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827690.342451:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827690/real 1787827690] req@ffff8ab3f04383c0 x1874671280668416/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827690.342460:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280668416/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827690.342469:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827690.342473:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827690.342477:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280668416:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827691.366037:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827692.390067:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827693.414112:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827694.438044:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827695.462000:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827695.462011:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827695.462017:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827695.462028:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2245 00000100:00080000:0.0:1787827695.462039:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827695.462048:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827695.462051:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827695.462088:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280668544/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827695.462116:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827695.462120:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827695.462122:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827695.462126:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2245 00000100:00080000:0.0:1787827695.462129:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827695.462132:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827695.462143:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827695.462156:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280668672/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827695.462168:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827695.462174:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787827695.462174:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280668544:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827695.462179:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827695.462191:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2245 00000100:00080000:0.0:1787827695.462194:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827695.462198:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827695.462203:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827695.462215:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280668800/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827695.462224:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827695.462228:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827695.462230:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827695.462233:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2245 00000100:00080000:0.0:1787827695.462246:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827695.462251:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:3.0:1787827695.462258:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787827695.462262:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827695.462278:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280668928/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827695.462288:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827695.462292:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827695.462293:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827695.462297:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2245 00000100:00080000:0.0:1787827695.462301:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827695.462305:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827695.462309:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827695.462321:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280669056/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827695.462330:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827695/real 1787827695] req@ffff8ab3f075cf00 x1874671280668544/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827695.462386:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280668544/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827695.462399:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827695.462403:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827695.462407:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280668544:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827695.462432:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280668672:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827695.462455:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827695.462468:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827695/real 1787827695] req@ffff8ab3f075f480 x1874671280668672/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827695.462480:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280668672/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827695.462488:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827695.462491:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827695.462494:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280668672:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827695.462502:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280668800:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827695.462517:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827695.462525:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827695/real 1787827695] req@ffff8ab3f075ed00 x1874671280668800/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827695.462536:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280668800/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827695.462543:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827695.462545:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827695.462548:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280668800:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827695.462554:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280668928:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827695.462580:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827695.462626:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827695/real 1787827695] req@ffff8ab3f075e580 x1874671280668928/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827695.462643:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280668928/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827695.462653:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827695.462656:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827695.462659:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280668928:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827695.462666:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280669056:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827695.462713:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827695.462731:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827695/real 1787827695] req@ffff8ab3f075cb40 x1874671280669056/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827695.462743:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280669056/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827695.462753:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827695.462755:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827695.462758:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280669056:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827696.486068:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827697.510056:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827698.534079:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827699.558066:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827700.581992:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827700.582000:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827700.582004:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827700.582013:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2250 00000100:00080000:2.0:1787827700.582022:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827700.582029:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787827700.582056:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827700.582064:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280669184/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827700.582086:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827700.582089:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827700.582090:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827700.582093:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2250 00000100:00080000:2.0:1787827700.582096:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827700.582100:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827700.582109:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827700.582118:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280669312/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827700.582126:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827700.582132:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787827700.582133:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280669184:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827700.582134:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827700.582142:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2250 00000100:00080000:2.0:1787827700.582151:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827700.582157:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827700.582160:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827700.582173:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280669440/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827700.582179:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827700.582193:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827700.582194:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827700.582197:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2250 00000100:00080000:2.0:1787827700.582199:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827700.582202:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827700.582205:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827700.582213:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280669568/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827700.582219:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827700.582222:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827700.582223:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827700.582226:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2250 00000100:00080000:2.0:1787827700.582261:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827700.582264:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827700.582267:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827700.582282:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280669696/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827700.582283:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827700.582356:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827700/real 1787827700] req@ffff8ab3f04392c0 x1874671280669184/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827700.582377:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280669184/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827700.582405:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827700.582409:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827700.582413:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280669184:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827700.582451:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280669312:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827700.582475:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827700.582487:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827700/real 1787827700] req@ffff8ab3f0439a40 x1874671280669312/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827700.582504:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280669312/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827700.582529:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827700.582533:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827700.582537:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280669312:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827700.582544:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280669440:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827700.582564:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827700.582574:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827700/real 1787827700] req@ffff8ab3f043a1c0 x1874671280669440/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827700.582587:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280669440/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827700.582595:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827700.582598:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827700.582602:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280669440:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827700.582608:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280669568:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827700.582625:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827700.582634:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827700/real 1787827700] req@ffff8ab3f043a940 x1874671280669568/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827700.582646:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280669568/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827700.582706:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827700.582709:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827700.582712:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280669568:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827700.582719:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280669696:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827700.582739:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827700.582749:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827700/real 1787827700] req@ffff8ab3f043b480 x1874671280669696/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827700.582759:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280669696/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827700.582768:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827700.582772:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827700.582774:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280669696:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827701.606063:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827702.630166:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827703.654076:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827704.678131:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827705.702064:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827705.702085:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827705.702097:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827705.702104:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827705.702118:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2255 00000100:00080000:3.0:1787827705.702134:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827705.702145:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827705.702219:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280669824/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827705.702269:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827705.702274:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827705.702276:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827705.702281:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2255 00000100:00080000:3.0:1787827705.702286:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827705.702293:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827705.702307:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827705.702320:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280669824:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827705.702326:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280669952/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827705.702346:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827705.702352:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827705.702354:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827705.702360:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2255 00000100:00080000:3.0:1787827705.702365:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827705.702370:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827705.702376:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827705.702392:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280670080/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827705.702405:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827705.702424:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827705.702427:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827705.702433:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827705.702444:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2255 00000100:00080000:3.0:1787827705.702452:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827705.702473:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827705.702482:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827705.702513:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280670208/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827705.702525:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827705/real 1787827705] req@ffff8ab3d0a421c0 x1874671280669824/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827705.702560:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827705.702570:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827705.702575:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827705.702579:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280669824/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827705.702595:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2255 00000100:00080000:3.0:1787827705.702633:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827705.702637:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.702650:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787827705.702652:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827705.702674:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827705.702684:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280669824:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827705.702697:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280670336/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827705.702720:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280669952:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827705.702753:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827705.702786:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827705/real 1787827705] req@ffff8ab3d0a403c0 x1874671280669952/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.702802:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280669952/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.702811:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.702815:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827705.702818:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280669952:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827705.702825:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280670080:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827705.702844:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827705.702854:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827705/real 1787827705] req@ffff8ab3d0a41e00 x1874671280670080/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.702866:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280670080/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.702874:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.702877:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827705.702880:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280670080:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827705.702886:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280670208:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827705.702953:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827705.702964:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827705/real 1787827705] req@ffff8ab3d0a412c0 x1874671280670208/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.702975:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280670208/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.702990:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.702993:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827705.702995:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280670208:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827705.703019:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280670336:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827705.703041:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827705.703051:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827705/real 1787827705] req@ffff8ab3d0a43c00 x1874671280670336/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.703063:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280670336/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.703073:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.703076:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827705.703079:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280670336:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827706.726003:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827707.750075:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827708.774043:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827709.798089:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827710.822037:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827710.822056:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827710.822063:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827710.822079:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2260 00000100:00080000:2.0:1787827710.822095:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827710.822106:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787827710.822114:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827710.822182:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280670464/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827710.822260:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827710.822265:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827710.822269:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827710.822277:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2260 00000100:00080000:2.0:1787827710.822284:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827710.822291:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827710.822306:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280670464:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787827710.822307:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827710.822333:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280670592/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827710.822350:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827710.822355:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827710.822359:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827710.822366:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2260 00000100:00080000:2.0:1787827710.822383:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827710.822392:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827710.822398:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827710.822425:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827710.822449:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280670720/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827710.822461:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827710/real 1787827710] req@ffff8ab3f0439e00 x1874671280670464/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827710.822462:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827710.822496:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827710.822500:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827710.822508:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280670464/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827710.822514:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2260 00000100:00080000:2.0:1787827710.822525:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827710.822541:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827710.822562:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:3.0:1787827710.822563:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827710.822595:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827710.822602:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280670464:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827710.822642:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280670848/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827710.822658:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827710.822664:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280670592:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787827710.822666:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827710.822701:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827710.822714:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2260 00000100:00080000:2.0:1787827710.822730:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827710.822731:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827710.822747:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827710.822756:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827710/real 1787827710] req@ffff8ab3f043b840 x1874671280670592/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827710.822760:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827710.822782:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280670592/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827710.822790:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280670976/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827710.822799:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827710.822804:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827710.822812:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280670592:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827710.822841:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280670720:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827710.822867:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827710.822887:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827710/real 1787827710] req@ffff8ab3f043b480 x1874671280670720/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827710.822901:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280670720/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827710.822938:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827710.822941:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827710.822943:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280670720:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827710.822956:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280670848:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827710.822972:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827710.822980:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827710/real 1787827710] req@ffff8ab3f043a940 x1874671280670848/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827710.822988:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280670848/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827710.822995:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827710.822997:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827710.822999:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280670848:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827710.823003:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280670976:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827710.823013:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827710.823018:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827710/real 1787827710] req@ffff8ab3f043a1c0 x1874671280670976/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827710.823025:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280670976/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827710.823031:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827710.823033:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827710.823035:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280670976:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827711.846086:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827712.869995:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827713.894059:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827714.918061:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827715.942101:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827715.942126:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827715.942137:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827715.942142:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827715.942152:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2265 00000100:00080000:3.0:1787827715.942161:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827715.942168:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827715.942206:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280671104/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827715.942241:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827715.942246:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827715.942249:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827715.942254:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2265 00000100:00080000:3.0:1787827715.942259:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827715.942265:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827715.942279:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827715.942300:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280671232/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827715.942321:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827715.942323:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280671104:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827715.942328:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827715.942344:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827715.942360:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2265 00000100:00080000:3.0:1787827715.942367:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827715.942373:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827715.942381:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827715.942405:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280671360/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827715.942417:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827715.942422:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827715.942424:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827715.942428:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2265 00000100:00080000:3.0:1787827715.942483:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827715.942491:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827715.942498:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787827715.942505:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827715.942528:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280671488/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827715.942561:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827715.942565:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827715.942567:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827715.942575:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2265 00000100:00080000:3.0:1787827715.942590:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827715.942594:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827715.942600:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827715.942617:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280671616/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827715.942657:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827715/real 1787827715] req@ffff8ab3d0a40000 x1874671280671104/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827715.942694:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280671104/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827715.942725:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827715.942733:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827715.942740:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280671104:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827715.942785:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280671232:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827715.942831:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827715.942854:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827715/real 1787827715] req@ffff8ab3d0a43480 x1874671280671232/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827715.942876:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280671232/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827715.942889:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827715.942896:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827715.942939:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280671232:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827715.942952:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280671360:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827715.942986:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827715.943005:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827715/real 1787827715] req@ffff8ab3d0a43840 x1874671280671360/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827715.943027:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280671360/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827715.943042:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827715.943048:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827715.943054:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280671360:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827715.943065:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280671488:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827715.943092:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827715.943108:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827715/real 1787827715] req@ffff8ab3d0a430c0 x1874671280671488/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827715.943128:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280671488/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827715.943144:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827715.943150:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827715.943156:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280671488:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827715.943166:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280671616:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827715.943192:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827715.943207:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827715/real 1787827715] req@ffff8ab3d0a43c00 x1874671280671616/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827715.943226:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280671616/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827715.943243:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827715.943249:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827715.943254:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280671616:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827716.966082:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827717.990119:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827719.014051:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827720.038067:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827721.061992:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827721.062003:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827721.062008:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827721.062018:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2270 00000100:00080000:3.0:1787827721.062029:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827721.062037:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827721.062050:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827721.062075:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280671744/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827721.062101:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827721.062105:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827721.062107:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827721.062112:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2270 00000100:00080000:3.0:1787827721.062117:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827721.062123:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827721.062129:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280671744:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787827721.062136:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827721.062151:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280671872/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827721.062161:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827721.062163:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827721.062165:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827721.062168:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2270 00000100:00080000:3.0:1787827721.062180:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827721.062185:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827721.062193:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827721.062197:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827721.062207:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280672000/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827721.062219:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827721.062221:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827721.062223:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827721.062226:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2270 00000100:00080000:3.0:1787827721.062229:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827721.062230:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827721/real 1787827721] req@ffff8ab3d0a412c0 x1874671280671744/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827721.062235:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827721.062245:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827721.062255:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280671744/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827721.062259:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280672128/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827721.062279:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827721.062280:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827721.062287:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827721.062288:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787827721.062289:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827721.062295:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280671744:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827721.062296:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2270 00000100:00080000:3.0:1787827721.062303:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827721.062307:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827721.062310:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827721.062317:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280672256/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827721.062326:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280671872:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827721.062354:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827721.062368:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827721/real 1787827721] req@ffff8ab3d0a41e00 x1874671280671872/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827721.062378:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280671872/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827721.062383:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827721.062386:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827721.062388:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280671872:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827721.062393:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280672000:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827721.062406:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827721.062412:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827721/real 1787827721] req@ffff8ab3d0a403c0 x1874671280672000/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827721.062420:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280672000/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827721.062425:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827721.062427:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827721.062429:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280672000:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827721.062433:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280672128:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827721.062443:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827721.062448:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827721/real 1787827721] req@ffff8ab3d0a421c0 x1874671280672128/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827721.062455:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280672128/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827721.062461:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827721.062463:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827721.062465:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280672128:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827721.062476:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280672256:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827721.062486:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827721.062492:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827721/real 1787827721] req@ffff8ab3d0a43c00 x1874671280672256/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827721.062499:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280672256/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827721.062527:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827721.062530:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827721.062532:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280672256:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827722.086089:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827723.110051:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827724.134104:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827725.158093:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827726.181971:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827726.181981:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827726.181985:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827726.182005:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2275 00000100:00080000:3.0:1787827726.182016:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827726.182024:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827726.182055:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827726.182064:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280672384/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827726.182091:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827726.182095:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827726.182097:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827726.182100:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2275 00000100:00080000:3.0:1787827726.182104:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827726.182108:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827726.182119:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827726.182127:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280672384:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827726.182141:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280672512/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827726.182152:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827726.182155:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827726.182157:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827726.182161:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2275 00000100:00080000:3.0:1787827726.182165:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827726.182168:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827726.182173:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827726.182184:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280672640/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827726.182193:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827726.182197:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827726.182199:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827726.182206:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827726.182207:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2275 00000100:00080000:3.0:1787827726.182216:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827726.182221:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827726.182226:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827726.182240:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280672768/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827726.182250:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827726.182254:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827726.182257:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827726.182263:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827726/real 1787827726] req@ffff8ab3d0a430c0 x1874671280672384/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827726.182270:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2275 00000100:00080000:3.0:1787827726.182283:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827726.182293:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280672384/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827726.182295:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827726.182308:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:2.0:1787827726.182317:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827726.182327:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827726.182332:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280672896/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827726.182339:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280672384:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827726.182381:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280672512:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827726.182407:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827726.182421:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827726/real 1787827726] req@ffff8ab3d0a43840 x1874671280672512/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827726.182433:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280672512/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827726.182442:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827726.182445:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827726.182448:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280672512:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827726.182454:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280672640:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827726.182468:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827726.182499:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827726/real 1787827726] req@ffff8ab3d0a43480 x1874671280672640/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827726.182511:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280672640/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827726.182519:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827726.182522:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827726.182525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280672640:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827726.182530:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280672768:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827726.182549:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827726.182557:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827726/real 1787827726] req@ffff8ab3d0a40000 x1874671280672768/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827726.182568:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280672768/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827726.182576:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827726.182578:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827726.182582:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280672768:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827726.182587:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280672896:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827726.182600:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827726.182607:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827726/real 1787827726] req@ffff8ab3d0a43c00 x1874671280672896/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827726.182617:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280672896/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827726.182625:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827726.182627:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827726.182630:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280672896:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827727.206215:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827728.230049:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827729.254049:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827730.277996:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827731.301986:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827731.301996:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827731.302001:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827731.302010:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2280 00000100:00080000:1.0:1787827731.302019:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827731.302025:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827731.302040:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827731.302060:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280673024/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827731.302082:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827731.302086:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827731.302088:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827731.302093:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2280 00000100:00080000:1.0:1787827731.302097:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827731.302103:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827731.302114:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827731.302116:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280673024:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787827731.302133:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280673152/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827731.302143:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827731.302146:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827731.302148:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827731.302150:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2280 00000100:00080000:1.0:1787827731.302170:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827731.302174:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827731.302178:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827731.302183:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787827731.302190:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280673280/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827731.302199:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827731.302201:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827731.302203:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827731.302206:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2280 00000100:00080000:1.0:1787827731.302208:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827731.302211:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827731.302216:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827731.302226:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827731/real 1787827731] req@ffff8ab3d22fed00 x1874671280673024/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827731.302231:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280673408/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827731.302268:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827731.302272:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280673024/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827731.302276:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827731.302282:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827731.302288:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827731.302293:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2280 00000100:00080000:2.0:1787827731.302297:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:1.0:1787827731.302300:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827731.302304:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280673024:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787827731.302311:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787827731.302318:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787827731.302330:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280673536/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827731.302336:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280673152:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827731.302384:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827731.302398:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827731/real 1787827731] req@ffff8ab3d22fd2c0 x1874671280673152/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827731.302408:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280673152/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827731.302415:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827731.302418:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827731.302421:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280673152:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827731.302426:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280673280:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827731.302464:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827731.302474:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827731/real 1787827731] req@ffff8ab3d22fde00 x1874671280673280/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827731.302498:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280673280/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827731.302504:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827731.302506:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827731.302508:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280673280:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827731.302511:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280673408:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827731.302523:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827731.302529:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827731/real 1787827731] req@ffff8ab3d22fe1c0 x1874671280673408/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827731.302536:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280673408/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827731.302543:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827731.302545:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827731.302546:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280673408:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827731.302557:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280673536:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827731.302567:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827731.302572:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827731/real 1787827731] req@ffff8ab3d22fc780 x1874671280673536/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827731.302579:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280673536/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827731.302585:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827731.302587:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827731.302589:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280673536:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827732.326014:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827733.350022:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827734.374018:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827735.398110:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827736.422049:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827736.422074:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827736.422079:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1787827736.422086:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827736.422089:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2286 00000100:00080000:2.0:1787827736.422099:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827736.422107:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827736.422145:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280673664/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827736.422176:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827736.422181:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827736.422183:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827736.422186:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2286 00000100:00080000:2.0:1787827736.422189:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827736.422193:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827736.422201:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827736.422231:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280673792/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827736.422238:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280673664:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827736.422244:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827736.422249:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827736.422251:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827736.422254:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2286 00000100:00080000:2.0:1787827736.422258:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827736.422262:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827736.422266:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827736.422294:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280673920/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827736.422309:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827736.422312:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827736.422313:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827736.422316:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2286 00000100:00080000:2.0:1787827736.422320:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827736.422325:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827736.422330:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827736.422352:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827736.422365:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280674048/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827736.422373:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827736.422377:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827736.422379:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827736.422384:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2286 00000100:00080000:2.0:1787827736.422389:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827736.422394:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827736.422398:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827736.422439:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827736/real 1787827736] req@ffff8ab3f0439a40 x1874671280673664/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827736.422445:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280674176/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827736.422468:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280673664/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827736.422478:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827736.422481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827736.422484:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280673664:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827736.422529:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280673792:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827736.422549:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827736.422560:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827736/real 1787827736] req@ffff8ab3f04392c0 x1874671280673792/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827736.422569:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280673792/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827736.422576:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827736.422578:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827736.422581:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280673792:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827736.422585:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280673920:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827736.422596:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827736.422602:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827736/real 1787827736] req@ffff8ab3f04383c0 x1874671280673920/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827736.422610:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280673920/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827736.422616:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827736.422618:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827736.422621:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280673920:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827736.422626:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280674048:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827736.422637:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827736.422643:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827736/real 1787827736] req@ffff8ab3f043bc00 x1874671280674048/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827736.422674:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280674048/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827736.422688:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827736.422692:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827736.422696:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280674048:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827736.422705:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280674176:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827736.422753:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827736.422762:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827736/real 1787827736] req@ffff8ab3f0439680 x1874671280674176/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827736.422770:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280674176/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827736.422777:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827736.422779:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827736.422781:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280674176:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827737.446062:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827738.470113:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827739.494096:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827740.518048:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827741.542045:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827741.542056:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827741.542062:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827741.542074:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2291 00000100:00080000:2.0:1787827741.542086:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827741.542095:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787827741.542132:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827741.542140:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438000 x1874671280674304/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827741.542181:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827741.542185:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827741.542187:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827741.542191:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2291 00000100:00080000:2.0:1787827741.542195:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827741.542201:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827741.542213:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827741.542227:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280674432/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827741.542240:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280674304:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827741.542247:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827741.542253:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827741.542256:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827741.542262:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2291 00000100:00080000:2.0:1787827741.542267:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827741.542273:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827741.542279:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827741.542297:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280674560/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827741.542323:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827741.542327:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827741.542328:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827741.542333:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2291 00000100:00080000:2.0:1787827741.542347:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827741.542353:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827741.542359:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:3.0:1787827741.542360:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827741.542400:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280674688/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827741.542443:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827741.542447:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827741.542450:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827741.542455:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2291 00000100:00080000:2.0:1787827741.542460:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827741.542466:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827741.542473:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827741.542514:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280674816/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827741.542585:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827741/real 1787827741] req@ffff8ab3f0438000 x1874671280674304/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827741.542637:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438000 x1874671280674304/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827741.542663:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827741.542672:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827741.542679:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280674304:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827741.542716:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280674432:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827741.542782:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827741.542819:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827741/real 1787827741] req@ffff8ab3f0438b40 x1874671280674432/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827741.542847:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280674432/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827741.542865:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827741.542871:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827741.542942:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280674432:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827741.542950:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280674560:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827741.542975:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827741.543003:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827741/real 1787827741] req@ffff8ab3f043b0c0 x1874671280674560/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827741.543016:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280674560/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827741.543024:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827741.543028:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827741.543031:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280674560:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827741.543042:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280674688:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827741.543060:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827741.543070:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827741/real 1787827741] req@ffff8ab3f0438780 x1874671280674688/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827741.543081:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280674688/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827741.543091:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827741.543093:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827741.543096:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280674688:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827741.543102:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280674816:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827741.543116:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827741.543124:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827741/real 1787827741] req@ffff8ab3f043a580 x1874671280674816/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827741.543134:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280674816/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827741.543151:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827741.543156:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827741.543162:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280674816:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827742.566083:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827743.590051:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827744.614078:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827745.638072:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827746.662002:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827746.662010:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827746.662013:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827746.662022:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2296 00000100:00080000:2.0:1787827746.662032:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827746.662038:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787827746.662056:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827746.662072:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280674944/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827746.662093:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827746.662095:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827746.662097:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827746.662099:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2296 00000100:00080000:2.0:1787827746.662102:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827746.662105:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827746.662114:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827746.662119:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280674944:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827746.662126:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280675072/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827746.662136:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827746.662138:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827746.662139:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827746.662142:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2296 00000100:00080000:2.0:1787827746.662145:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827746.662147:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827746.662150:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827746.662159:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280675200/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827746.662166:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827746.662169:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827746.662170:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787827746.662175:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827746.662177:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2296 00000100:00080000:2.0:1787827746.662182:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827746.662185:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827746.662189:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827746.662209:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280675328/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827746.662216:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827746.662219:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827746.662220:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827746.662223:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2296 00000100:00080000:2.0:1787827746.662225:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827746.662228:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827746.662232:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827746.662239:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827746/real 1787827746] req@ffff8ab3f0438f00 x1874671280674944/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827746.662242:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280675456/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827746.662286:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280674944/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827746.662295:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827746.662298:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827746.662302:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280674944:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827746.662320:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280675072:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827746.662351:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827746.662379:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827746/real 1787827746] req@ffff8ab3f043ad00 x1874671280675072/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827746.662388:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280675072/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827746.662395:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827746.662397:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827746.662400:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280675072:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827746.662404:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280675200:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827746.662417:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827746.662423:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827746/real 1787827746] req@ffff8ab3f043a580 x1874671280675200/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827746.662431:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280675200/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827746.662436:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827746.662438:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827746.662440:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280675200:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827746.662444:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280675328:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827746.662455:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827746.662461:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827746/real 1787827746] req@ffff8ab3f0438780 x1874671280675328/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827746.662469:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280675328/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827746.662476:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827746.662478:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827746.662480:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280675328:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827746.662484:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280675456:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827746.662494:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827746.662500:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827746/real 1787827746] req@ffff8ab3f043b0c0 x1874671280675456/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827746.662516:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280675456/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827746.662522:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827746.662524:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827746.662526:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280675456:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827747.686168:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827748.710049:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827749.734027:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827750.758083:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827751.782009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827751.782019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827751.782023:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827751.782032:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2301 00000100:00080000:3.0:1787827751.782041:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827751.782062:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827751.782071:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827751.782097:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280675584/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827751.782136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827751.782141:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827751.782143:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827751.782147:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2301 00000100:00080000:3.0:1787827751.782150:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827751.782154:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827751.782163:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280675584:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787827751.782164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827751.782178:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280675712/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827751.782202:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827751.782204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827751.782206:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827751.782209:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2301 00000100:00080000:3.0:1787827751.782211:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827751.782214:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827751.782217:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827751.782225:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280675840/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827751.782232:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827751.782234:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827751.782236:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827751.782238:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2301 00000100:00080000:3.0:1787827751.782241:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827751.782244:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827751.782251:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827751.782266:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280675968/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827751.782276:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827751.782281:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787827751.782282:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280675712:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787827751.782285:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827751.782303:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2301 00000100:00080000:3.0:1787827751.782308:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827751.782312:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827751.782319:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827751.782334:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280675840:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827751.782335:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280676096/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827751.782358:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280675968:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827751.782378:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280676096:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827752.806104:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827753.830068:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827754.854133:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827754.854222:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787827754.854310:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827754.854331:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827754.854379:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827754.854397:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827754.854415:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827754.854432:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:2.0:1787827754.854438:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827751/real 1787827754] req@ffff8ab3d0a403c0 x1874671280675712/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827754.854476:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280675712/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827754.854488:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827754.854493:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827754.854498:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280675712:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827754.854506:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827751/real 1787827754] req@ffff8ab3d0a41e00 x1874671280675840/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827754.854518:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280675840/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827754.854525:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827754.854528:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827754.854532:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280675840:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827754.854539:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827751/real 1787827754] req@ffff8ab3d0a412c0 x1874671280675968/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827754.854556:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280675968/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827754.854571:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827754.854575:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827754.854579:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280675968:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827754.854585:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827751/real 1787827754] req@ffff8ab3d0a421c0 x1874671280675584/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827754.854597:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280675584/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827754.854604:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827754.854607:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827754.854610:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280675584:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827754.854638:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827751/real 1787827754] req@ffff8ab3d0a43c00 x1874671280676096/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827754.854651:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280676096/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827754.854659:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827754.854662:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827754.854665:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280676096:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827755.878079:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827756.902008:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827756.902011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827756.902018:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827756.902022:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827756.902030:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2306 00000100:00080000:0.0:1787827756.902038:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827756.902044:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827756.902078:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280676224/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827756.902106:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827756.902110:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827756.902112:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827756.902115:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2306 00000100:00080000:0.0:1787827756.902119:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827756.902123:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827756.902131:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827756.902143:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280676224:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827756.902145:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280676352/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827756.902158:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827756.902161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827756.902162:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827756.902165:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2306 00000100:00080000:0.0:1787827756.902168:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827756.902170:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827756.902173:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827756.902181:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280676480/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827756.902188:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827756.902191:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827756.902192:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827756.902195:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2306 00000100:00080000:0.0:1787827756.902201:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827756.902206:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827756.902211:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787827756.902214:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827756.902226:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280676608/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827756.902237:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827756.902240:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827756.902242:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827756.902246:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2306 00000100:00080000:0.0:1787827756.902271:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827756.902277:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827756.902283:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827756/real 1787827756] req@ffff8ab3f075da40 x1874671280676224/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827756.902284:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827756.902302:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280676736/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827756.902309:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280676224/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827756.902326:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827756.902331:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827756.902336:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280676224:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827756.902361:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280676352:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827756.902385:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827756.902401:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827756/real 1787827756] req@ffff8ab3f075cb40 x1874671280676352/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827756.902414:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280676352/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827756.902421:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827756.902425:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827756.902428:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280676352:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827756.902434:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280676480:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827756.902454:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827756.902467:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827756/real 1787827756] req@ffff8ab3f075e580 x1874671280676480/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827756.902481:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280676480/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827756.902508:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827756.902512:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827756.902516:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280676480:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827756.902523:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280676608:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827756.902543:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827756.902558:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827756/real 1787827756] req@ffff8ab3f075ed00 x1874671280676608/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827756.902569:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280676608/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827756.902579:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827756.902582:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827756.902585:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280676608:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827756.902592:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280676736:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827756.902609:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827756.902620:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827756/real 1787827756] req@ffff8ab3f075f480 x1874671280676736/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827756.902632:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280676736/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827756.902642:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827756.902644:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827756.902648:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280676736:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827757.926117:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827758.950131:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827759.974024:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827760.998024:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827762.022033:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1787827762.022041:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827762.022045:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827762.022051:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827762.022064:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2311 00000100:00080000:3.0:1787827762.022075:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827762.022083:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827762.022131:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280676864/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827762.022180:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827762.022185:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827762.022188:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827762.022196:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2311 00000100:00080000:3.0:1787827762.022202:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827762.022208:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827762.022234:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280676864:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787827762.022243:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827762.022265:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280676992/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827762.022297:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827762.022301:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827762.022304:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827762.022308:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2311 00000100:00080000:3.0:1787827762.022312:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827762.022319:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827762.022323:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827762.022331:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827762.022346:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280677120/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827762.022355:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827762.022358:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827762.022360:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827762.022364:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2311 00000100:00080000:3.0:1787827762.022367:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827762.022371:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827762/real 1787827762] req@ffff8ab3d0a40000 x1874671280676864/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827762.022376:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827762.022389:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827762.022404:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280676864/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827762.022408:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280677248/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827762.022453:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827762.022456:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827762.022461:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787827762.022464:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827762.022469:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827762.022471:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280676864:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827762.022481:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2311 00000100:00080000:3.0:1787827762.022487:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827762.022491:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827762.022495:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827762.022511:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280676992:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827762.022511:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280677376/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827762.022546:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827762.022560:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827762/real 1787827762] req@ffff8ab3d0a43480 x1874671280676992/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827762.022574:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280676992/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827762.022584:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827762.022587:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827762.022597:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280676992:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827762.022605:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280677120:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827762.022626:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827762.022639:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827762/real 1787827762] req@ffff8ab3d0a43840 x1874671280677120/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827762.022650:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280677120/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827762.022657:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827762.022660:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827762.022663:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280677120:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827762.022678:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280677248:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827762.022698:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827762.022709:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827762/real 1787827762] req@ffff8ab3d0a430c0 x1874671280677248/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827762.022721:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280677248/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827762.022737:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827762.022741:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827762.022744:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280677248:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827762.022761:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280677376:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827762.022783:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827762.022796:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827762/real 1787827762] req@ffff8ab3d0a40000 x1874671280677376/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827762.022808:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280677376/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827762.022818:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827762.022822:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827762.022825:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280677376:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827763.046079:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827764.070048:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827765.094081:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827766.118051:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827767.142011:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827767.142022:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827767.142026:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827767.142036:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2316 00000100:00080000:2.0:1787827767.142048:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1787827767.142067:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827767.142069:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827767.142105:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280677504/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827767.142139:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827767.142142:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827767.142143:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827767.142146:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2316 00000100:00080000:2.0:1787827767.142150:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827767.142154:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827767.142163:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827767.142166:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280677504:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827767.142174:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438000 x1874671280677632/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827767.142182:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827767.142196:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827767.142199:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827767.142203:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2316 00000100:00080000:2.0:1787827767.142206:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827767.142209:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827767.142223:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827767.142233:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280677760/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827767.142240:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827767.142242:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827767.142244:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827767.142247:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2316 00000100:00080000:2.0:1787827767.142251:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827767.142252:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827767.142257:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827767.142261:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827767.142269:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280677888/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827767.142275:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827767.142278:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827767.142279:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827767.142282:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2316 00000100:00080000:2.0:1787827767.142285:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827767.142287:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827767.142290:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827767.142298:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280678016/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827767.142306:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827767/real 1787827767] req@ffff8ab3f0438b40 x1874671280677504/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827767.142323:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280677504/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827767.142331:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827767.142334:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827767.142336:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280677504:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827767.142355:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280677632:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827767.142373:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827767.142382:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827767/real 1787827767] req@ffff8ab3f0438000 x1874671280677632/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827767.142390:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438000 x1874671280677632/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827767.142395:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827767.142397:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827767.142400:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280677632:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827767.142404:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280677760:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827767.142414:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827767.142429:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827767/real 1787827767] req@ffff8ab3f0439680 x1874671280677760/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827767.142436:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280677760/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827767.142441:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827767.142442:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827767.142444:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280677760:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827767.142448:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280677888:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827767.142457:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827767.142463:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827767/real 1787827767] req@ffff8ab3f043bc00 x1874671280677888/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827767.142470:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280677888/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827767.142476:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827767.142478:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827767.142480:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280677888:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827767.142483:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280678016:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827767.142499:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827767.142504:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827767/real 1787827767] req@ffff8ab3f04383c0 x1874671280678016/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827767.142511:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280678016/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827767.142518:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827767.142519:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827767.142521:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280678016:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827768.166041:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827769.190053:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827770.214064:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827771.238010:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827772.261994:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827772.262003:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827772.262007:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827772.262016:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2321 00010000:00010000:2.0:1787827772.262019:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827772.262024:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827772.262031:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827772.262064:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280678144/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827772.262087:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827772.262090:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827772.262091:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827772.262094:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2321 00000100:00080000:0.0:1787827772.262096:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827772.262098:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827772.262106:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827772.262114:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280678272/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827772.262122:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827772.262126:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827772.262128:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827772.262133:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2321 00000100:00080000:0.0:1787827772.262136:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827772.262136:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280678144:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827772.262141:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827772.262148:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827772.262171:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280678400/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827772.262180:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827772.262182:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827772.262184:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827772.262186:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2321 00000100:00080000:0.0:1787827772.262204:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827772.262208:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827772.262211:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:3.0:1787827772.262216:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827772.262221:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280678528/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827772.262229:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827772.262232:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827772.262233:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827772.262236:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2321 00000100:00080000:0.0:1787827772.262238:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827772.262240:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827772.262244:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827772.262253:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280678656/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827772.262269:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827772/real 1787827772] req@ffff8ab3f075cf00 x1874671280678144/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827772.262288:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280678144/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827772.262299:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827772.262304:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827772.262309:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280678144:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827772.262317:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280678272:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827772.262339:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827772.262361:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827772/real 1787827772] req@ffff8ab3f075f0c0 x1874671280678272/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827772.262374:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280678272/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827772.262384:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827772.262387:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827772.262391:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280678272:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827772.262413:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280678400:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827772.262433:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827772.262444:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827772/real 1787827772] req@ffff8ab3f075de00 x1874671280678400/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827772.262457:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280678400/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827772.262465:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827772.262468:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827772.262472:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280678400:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827772.262479:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280678528:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827772.262494:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827772.262504:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827772/real 1787827772] req@ffff8ab3f075f840 x1874671280678528/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827772.262517:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280678528/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827772.262527:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827772.262530:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827772.262533:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280678528:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827772.262539:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280678656:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827772.262556:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827772.262566:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827772/real 1787827772] req@ffff8ab3f075e940 x1874671280678656/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827772.262578:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280678656/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827772.262589:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827772.262592:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827772.262595:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280678656:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827773.286039:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827774.310159:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827775.334057:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827776.358178:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827777.382016:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827777.382028:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827777.382034:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827777.382047:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2327 00000100:00080000:2.0:1787827777.382058:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1787827777.382067:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827777.382068:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827777.382112:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280678784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827777.382160:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827777.382165:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827777.382167:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827777.382172:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2327 00000100:00080000:2.0:1787827777.382176:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827777.382181:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827777.382193:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827777.382209:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280678912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827777.382217:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280678784:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827777.382239:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827777.382245:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827777.382248:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827777.382255:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2327 00000100:00080000:2.0:1787827777.382259:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827777.382264:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827777.382270:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827777.382286:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280679040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827777.382297:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827777.382302:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1787827777.382307:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827777.382308:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827777.382318:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2327 00000100:00080000:2.0:1787827777.382323:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827777.382329:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827777.382337:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827777.382357:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280679168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827777.382368:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827777/real 1787827777] req@ffff8ab3f04392c0 x1874671280678784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827777.382375:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827777.382386:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827777.382389:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827777.382395:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280678784/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827777.382400:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2327 00000100:00080000:2.0:1787827777.382415:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827777.382421:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827777.382426:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:3.0:1787827777.382431:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827777.382441:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280678784:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787827777.382444:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827777.382466:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280679296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827777.382476:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280678912:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827777.382514:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827777.382532:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827777/real 1787827777] req@ffff8ab3f0439a40 x1874671280678912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827777.382542:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280678912/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827777.382548:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827777.382550:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827777.382553:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280678912:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827777.382566:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280679040:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827777.382582:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827777.382589:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827777/real 1787827777] req@ffff8ab3f043a1c0 x1874671280679040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827777.382596:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280679040/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827777.382606:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827777.382609:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827777.382611:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280679040:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827777.382615:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280679168:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827777.382627:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827777.382632:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827777/real 1787827777] req@ffff8ab3f043a940 x1874671280679168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827777.382640:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280679168/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827777.382647:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827777.382649:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827777.382651:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280679168:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827777.382666:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280679296:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827777.382686:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827777.382697:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827777/real 1787827777] req@ffff8ab3f043b480 x1874671280679296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827777.382711:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280679296/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827777.382720:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827777.382723:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827777.382728:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280679296:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827778.406051:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827779.430081:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827780.454117:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827781.478122:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827782.502073:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827782.502098:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827782.502102:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827782.502141:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2332 00010000:00010000:2.0:1787827782.502142:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827782.502151:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827782.502159:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827782.502205:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280679424/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827782.502228:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827782.502231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827782.502232:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827782.502235:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2332 00000100:00080000:3.0:1787827782.502237:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827782.502240:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827782.502248:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827782.502257:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280679552/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827782.502266:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280679424:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827782.502267:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827782.502274:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827782.502278:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827782.502282:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2332 00000100:00080000:3.0:1787827782.502284:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827782.502287:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827782.502301:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827782.502310:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280679680/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827782.502316:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827782.502319:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827782.502320:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827782.502323:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2332 00000100:00080000:3.0:1787827782.502337:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827782.502340:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827782.502343:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787827782.502350:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827782.502354:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280679808/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827782.502371:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827782.502374:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827782.502375:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827782.502378:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2332 00000100:00080000:3.0:1787827782.502381:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827782.502384:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827782.502387:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827782.502394:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280679936/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827782.502420:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827782/real 1787827782] req@ffff8ab3d0a43c00 x1874671280679424/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827782.502452:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280679424/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827782.502464:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827782.502468:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827782.502472:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280679424:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827782.502496:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280679552:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827782.502520:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827782.502536:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827782/real 1787827782] req@ffff8ab3d0a421c0 x1874671280679552/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827782.502550:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280679552/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827782.502559:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827782.502563:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827782.502566:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280679552:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827782.502573:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280679680:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827782.502610:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827782.502624:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827782/real 1787827782] req@ffff8ab3d0a412c0 x1874671280679680/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827782.502638:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280679680/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827782.502647:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827782.502651:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827782.502681:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280679680:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827782.502688:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280679808:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827782.502710:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827782.502723:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827782/real 1787827782] req@ffff8ab3d0a41e00 x1874671280679808/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827782.502741:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280679808/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827782.502752:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827782.502756:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827782.502759:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280679808:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827782.502766:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280679936:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827782.502789:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827782.502802:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827782/real 1787827782] req@ffff8ab3d0a403c0 x1874671280679936/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827782.502815:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280679936/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827782.502825:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827782.502829:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827782.502833:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280679936:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827783.526060:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827784.550137:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827785.574133:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827786.598117:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827787.622005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827787.622016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827787.622021:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827787.622034:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2337 00000100:00080000:0.0:1787827787.622059:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827787.622085:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827787.622119:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827787.622150:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280680064/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827787.622181:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827787.622184:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827787.622186:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827787.622191:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2337 00000100:00080000:0.0:1787827787.622195:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827787.622199:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827787.622210:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827787.622219:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280680064:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827787.622245:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280680192/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827787.622259:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827787.622262:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827787.622264:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827787.622268:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2337 00000100:00080000:0.0:1787827787.622272:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827787.622277:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827787.622282:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827787.622312:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280680320/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827787.622320:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787827787.622331:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827787.622335:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827787.622338:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827787.622343:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2337 00000100:00080000:0.0:1787827787.622349:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827787.622371:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827787.622379:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827787.622392:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827787/real 1787827787] req@ffff8ab3f075c000 x1874671280680064/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827787.622397:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280680448/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827787.622442:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827787.622443:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280680064/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827787.622463:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827787.622468:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827787.622469:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827787.622481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787827787.622482:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2337 00000100:00100000:2.0:1787827787.622492:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280680064:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827787.622493:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827787.622506:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827787.622514:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827787.622529:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280680576/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827787.622534:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280680192:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827787.622582:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827787.622604:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827787/real 1787827787] req@ffff8ab3f075e940 x1874671280680192/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827787.622627:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280680192/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827787.622635:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827787.622639:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827787.622642:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280680192:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827787.622649:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280680320:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827787.622670:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827787.622682:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827787/real 1787827787] req@ffff8ab3f075f840 x1874671280680320/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827787.622691:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280680320/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827787.622700:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827787.622703:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827787.622705:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280680320:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827787.622710:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280680448:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827787.622724:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827787.622730:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827787/real 1787827787] req@ffff8ab3f075de00 x1874671280680448/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827787.622737:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280680448/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827787.622744:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827787.622746:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827787.622748:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280680448:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827787.622760:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280680576:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827787.622780:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827787.622787:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827787/real 1787827787] req@ffff8ab3f075f0c0 x1874671280680576/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827787.622794:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280680576/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827787.622800:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827787.622802:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827787.622805:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280680576:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827788.646105:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827789.671175:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827790.694077:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827791.718038:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827792.741963:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827792.741970:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827792.741974:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827792.741982:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2342 00000100:00080000:0.0:1787827792.741991:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827792.741997:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827792.742018:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827792.742085:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280680704/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827792.742106:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827792.742109:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827792.742110:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827792.742113:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2342 00000100:00080000:0.0:1787827792.742117:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827792.742120:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827792.742128:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827792.742134:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280680704:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827792.742140:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280680832/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827792.742149:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827792.742151:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827792.742153:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827792.742155:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2342 00000100:00080000:0.0:1787827792.742157:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827792.742160:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827792.742163:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827792.742171:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280680960/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827792.742178:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827792.742181:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827792.742182:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827792.742185:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2342 00000100:00080000:0.0:1787827792.742195:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827792.742198:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827792.742201:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827792.742208:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280681088/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827792.742213:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787827792.742221:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827792.742224:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827792.742225:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827792.742229:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2342 00000100:00080000:0.0:1787827792.742234:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827792.742237:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827792.742240:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827792.742251:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280681216/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827792.742260:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827792/real 1787827792] req@ffff8ab3f075cf00 x1874671280680704/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827792.742275:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280680704/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827792.742284:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827792.742287:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827792.742290:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280680704:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827792.742317:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280680832:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827792.742333:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827792.742342:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827792/real 1787827792] req@ffff8ab3f075f480 x1874671280680832/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827792.742350:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280680832/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827792.742356:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827792.742358:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827792.742361:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280680832:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827792.742365:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280680960:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787827792.742375:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827792.742380:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827792/real 1787827792] req@ffff8ab3f075ed00 x1874671280680960/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827792.742387:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280680960/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827792.742392:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827792.742394:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827792.742396:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280680960:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787827792.742399:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280681088:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827792.742409:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827792.742414:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827792/real 1787827792] req@ffff8ab3f075e580 x1874671280681088/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827792.742421:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280681088/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827792.742427:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827792.742429:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827792.742431:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280681088:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787827792.742434:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280681216:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787827792.742443:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827792.742449:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827792/real 1787827792] req@ffff8ab3f075cb40 x1874671280681216/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827792.742456:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280681216/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827792.742462:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827792.742463:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787827792.742466:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280681216:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827793.766055:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827794.790132:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827795.814137:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827796.838095:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827797.862028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827797.862039:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827797.862044:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827797.862056:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2347 00010000:00010000:3.0:1787827797.862085:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827797.862126:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827797.862136:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827797.862184:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280681344/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827797.862213:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827797.862218:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827797.862220:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827797.862225:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2347 00000100:00080000:2.0:1787827797.862230:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827797.862237:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827797.862247:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827797.862252:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280681344:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827797.862263:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280681472/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827797.862278:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827797.862283:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827797.862285:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827797.862290:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2347 00000100:00080000:2.0:1787827797.862332:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827797.862338:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827797.862346:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827797.862350:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827797.862363:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280681600/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827797.862375:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827797.862380:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827797.862382:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827797.862388:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2347 00000100:00080000:2.0:1787827797.862393:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827797.862398:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827797/real 1787827797] req@ffff8ab3f043b840 x1874671280681344/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827797.862401:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827797.862410:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827797.862421:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280681344/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827797.862427:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280681728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827797.862434:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827797.862443:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787827797.862447:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827797.862449:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280681344:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827797.862458:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827797.862462:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827797.862467:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2347 00000100:00080000:2.0:1787827797.862471:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827797.862476:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280681472:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787827797.862478:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827797.862486:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827797.862498:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280681856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827797.862512:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827797.862535:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827797/real 1787827797] req@ffff8ab3f0439e00 x1874671280681472/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827797.862545:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280681472/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827797.862551:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827797.862553:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827797.862556:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280681472:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827797.862561:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280681600:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827797.862574:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827797.862580:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827797/real 1787827797] req@ffff8ab3f043b480 x1874671280681600/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827797.862588:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280681600/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827797.862594:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827797.862596:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827797.862598:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280681600:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827797.862601:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280681728:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827797.862612:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827797.862618:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827797/real 1787827797] req@ffff8ab3f043a940 x1874671280681728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827797.862625:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280681728/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827797.862632:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827797.862634:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827797.862636:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280681728:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827797.862646:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280681856:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827797.862656:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827797.862662:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827797/real 1787827797] req@ffff8ab3f043a1c0 x1874671280681856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827797.862674:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280681856/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827797.862681:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827797.862683:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827797.862685:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280681856:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827798.886083:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827799.910038:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827800.934097:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827801.958022:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827802.981992:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827802.982001:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827802.982005:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827802.982013:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2352 00000100:00080000:2.0:1787827802.982021:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827802.982026:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827802.982061:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280681984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787827802.982069:0:47651:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827802.982087:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827802.982091:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827802.982092:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827802.982095:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2352 00000100:00080000:2.0:1787827802.982097:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827802.982099:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827802.982119:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827802.982128:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280682112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827802.982143:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827802.982147:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827802.982148:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827802.982152:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280681984:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827802.982152:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2352 00000100:00080000:2.0:1787827802.982157:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827802.982161:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827802.982164:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827802.982174:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280682240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827802.982181:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827802.982184:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827802.982186:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827802.982188:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2352 00000100:00080000:2.0:1787827802.982202:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827802.982206:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827802.982209:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787827802.982213:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827802.982223:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280682368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827802.982231:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827802.982233:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827802.982235:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827802.982238:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2352 00000100:00080000:2.0:1787827802.982240:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827802.982242:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827802.982246:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827802.982256:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280682496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827802.982260:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827802/real 1787827802] req@ffff8ab3f0439a40 x1874671280681984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827802.982282:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280681984/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827802.982290:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827802.982294:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827802.982297:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280681984:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827802.982315:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280682112:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827802.982329:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827802.982337:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827802/real 1787827802] req@ffff8ab3f04392c0 x1874671280682112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827802.982348:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280682112/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827802.982356:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827802.982360:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827802.982363:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280682112:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827802.982380:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280682240:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827802.982401:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827802.982413:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827802/real 1787827802] req@ffff8ab3f04383c0 x1874671280682240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827802.982422:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280682240/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827802.982442:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827802.982444:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827802.982446:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280682240:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827802.982450:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280682368:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827802.982464:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827802.982470:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827802/real 1787827802] req@ffff8ab3f043bc00 x1874671280682368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827802.982478:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280682368/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827802.982484:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827802.982486:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827802.982488:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280682368:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827802.982491:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280682496:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827802.982500:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827802.982505:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827802/real 1787827802] req@ffff8ab3f0439680 x1874671280682496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827802.982512:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280682496/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827802.982519:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827802.982520:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827802.982523:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280682496:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827804.006029:0:47650:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827805.030258:0:27833:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 112/112 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1787827805.030306:0:47651:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 112 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827805.030320:0:47651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000022463a61/0x268e1a602cc1903 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800028400020000 nid: local remote: 0x772721ad1f0d81c3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827805.030495:0:27833:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 126/126 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1787827805.030539:0:47650:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 126 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787827805.030630:0:27833:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 81/81 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1787827805.031000:0:48572:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 81 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787827805.031037:0:47650:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f76f7717/0x268e1a602cc196c lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800028400020000 nid: local remote: 0x772721ad1f0d82d4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827805.031280:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827805.031286:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827805.031292:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827805.031308:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827805.031351:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x51:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031420:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 2/0,0 mode: CR/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da8dd expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827805.031465:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x50:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031477:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x4f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x772721ad1f0da8ac expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827805.031522:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854200/0x268e1a602cc2597 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da866 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031534:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x4e:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031541:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854600/0x268e1a602cc2574 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da80b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031551:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x4d:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031557:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854e00/0x268e1a602cc2551 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da7b0 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031573:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x4c:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031590:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031599:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x4b:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031605:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0970200/0x268e1a602cc250b lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da6fa expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031614:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x4a:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031619:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0970400/0x268e1a602cc24e8 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da69f expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031628:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x49:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031634:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0970600/0x268e1a602cc24c5 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da644 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031642:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x48:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031647:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0971e00/0x268e1a602cc24a2 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da5e9 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031656:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x47:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031661:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0970a00/0x268e1a602cc247f lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da58e expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031670:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x46:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031676:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0971c00/0x268e1a602cc245c lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da533 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031702:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x45:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031727:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2858000/0x268e1a602cc2439 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da4d8 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031740:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x44:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031749:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859a00/0x268e1a602cc2416 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da47d expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031763:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x43:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031773:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859000/0x268e1a602cc23f3 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da422 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031787:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x42:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031796:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859e00/0x268e1a602cc23d0 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da3c7 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031805:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x41:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031826:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859200/0x268e1a602cc23ad lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da36c expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031837:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x40:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031843:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2858a00/0x268e1a602cc238a lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da311 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031852:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x3f:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031858:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031872:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x3e:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031878:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031886:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x3d:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031892:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031901:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x3c:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031907:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031915:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x3b:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.031921:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.031995:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x3a:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032001:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032007:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x39:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032011:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032043:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x38:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032048:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032059:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x37:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032064:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032070:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x36:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032075:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032081:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x35:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032086:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032092:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x34:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032096:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032103:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x33:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032107:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032113:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x32:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032118:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032127:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x31:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032136:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032154:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x30:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032162:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032185:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x2f:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032193:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07600/0x268e1a602cc2137 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9d06 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032203:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x2e:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032209:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06c00/0x268e1a602cc2114 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9cab expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032216:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x2d:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032221:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07c00/0x268e1a602cc20f1 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9c50 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032227:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x2c:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032231:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07400/0x268e1a602cc20ce lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9bf5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032237:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x2b:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032242:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06800/0x268e1a602cc20ab lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9b9a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032248:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x2a:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032252:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859400/0x268e1a602cc2088 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9b3f expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032264:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x29:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032272:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08ea400/0x268e1a602cc2065 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9ae4 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032285:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x28:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032292:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eae00/0x268e1a602cc2042 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9a89 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032305:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x27:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032313:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0aa00/0x268e1a602cc201f lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9a2e expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032326:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x26:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032332:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3a600/0x268e1a602cc1ffc lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d99d3 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032340:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x25:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032344:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897800/0x268e1a602cc1fd9 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9978 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032362:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x24:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032366:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0b000/0x268e1a602cc1fb6 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d991d expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032374:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x23:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032382:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3be00/0x268e1a602cc1f93 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d98c2 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032401:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x22:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032408:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3a000/0x268e1a602cc1f70 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9867 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032421:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x21:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032429:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3b200/0x268e1a602cc1f4d lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d980c expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032450:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x20:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032455:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3b000/0x268e1a602cc1f2a lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d97b1 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032463:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x1f:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032467:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3ba00/0x268e1a602cc1f07 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9756 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032473:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x1e:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032478:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eaa00/0x268e1a602cc1ee4 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d96fb expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032485:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x1d:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032489:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eb600/0x268e1a602cc1ec1 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d96a0 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032507:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x1c:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032529:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08ea000/0x268e1a602cc1e9e lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9645 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032542:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x1b:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032547:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eac00/0x268e1a602cc1e7b lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d95ea expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032554:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x1a:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032559:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08ebc00/0x268e1a602cc1e58 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d958f expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032566:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x19:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032571:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eb400/0x268e1a602cc1e35 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9534 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032578:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x18:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032593:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4200/0x268e1a602cc1e12 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d94d9 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032600:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x17:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032606:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4800/0x268e1a602cc1def lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d947e expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032631:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x16:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032637:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5400/0x268e1a602cc1dcc lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9423 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032645:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x15:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032650:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4000/0x268e1a602cc1da9 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d93c8 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032663:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x14:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032669:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5200/0x268e1a602cc1d86 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d936d expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032677:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x13:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032682:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b7a00/0x268e1a602cc1d63 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9312 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032690:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x12:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032712:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0622c00/0x268e1a602cc1d40 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d92b7 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032724:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x11:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032733:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0623400/0x268e1a602cc1d1d lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d925c expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032745:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x10:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032754:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0623a00/0x268e1a602cc1cfa lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9201 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032767:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0xf:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032776:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0622800/0x268e1a602cc1cd7 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d91a6 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032789:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0xe:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032813:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0623200/0x268e1a602cc1cb4 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d914b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032828:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0xd:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032833:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0623e00/0x268e1a602cc1c91 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d90f0 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032842:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0xc:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032847:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c367ba00/0x268e1a602cc1c6e lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d9095 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032855:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0xb:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032861:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b7c00/0x268e1a602cc1c4b lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d903a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032868:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0xa:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032873:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b6600/0x268e1a602cc1c28 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8fdf expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032881:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x9:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032886:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b6800/0x268e1a602cc1c05 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8f84 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032894:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x8:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032899:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b6e00/0x268e1a602cc1be2 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8f29 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032907:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x7:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032912:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b6c00/0x268e1a602cc1bbf lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8ece expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032921:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x6:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.032986:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b7800/0x268e1a602cc1b9c lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8e73 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.032996:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x5:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.033001:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b7200/0x268e1a602cc1b79 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8e18 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.033008:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x4:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.033024:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf8a00/0x268e1a602cc1b56 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8dbd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.033031:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x3:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.033036:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc1b25 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8d38 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827805.033043:0:48572:0:(vvp_object.c:117:vvp_conf_set()) [0x240001b71:0x2:0x0]: losing layout lock 00010000:00010000:2.0:1787827805.033048:0:48572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958600/0x268e1a602cc1b41 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x772721ad1f0d8d77 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827805.033087:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827805.033094:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 1/0,0 mode: --/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da8dd expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827805.033101:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x4f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0da8ac expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827805.033107:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854200/0x268e1a602cc2597 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da866 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033118:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854600/0x268e1a602cc2574 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da80b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033127:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854e00/0x268e1a602cc2551 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da7b0 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033137:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033147:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0970200/0x268e1a602cc250b lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da6fa expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033168:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0970400/0x268e1a602cc24e8 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da69f expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033196:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0970600/0x268e1a602cc24c5 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da644 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033208:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0971e00/0x268e1a602cc24a2 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da5e9 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033220:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0970a00/0x268e1a602cc247f lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da58e expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033229:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0971c00/0x268e1a602cc245c lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da533 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033236:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2858000/0x268e1a602cc2439 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da4d8 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033244:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859a00/0x268e1a602cc2416 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da47d expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033251:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859000/0x268e1a602cc23f3 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da422 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033258:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859e00/0x268e1a602cc23d0 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da3c7 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033265:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859200/0x268e1a602cc23ad lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da36c expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033276:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2858a00/0x268e1a602cc238a lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da311 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033286:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033297:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033310:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033323:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033334:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033342:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033349:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033355:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033362:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033368:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033378:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033389:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033401:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033412:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033425:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033449:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033457:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07600/0x268e1a602cc2137 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9d06 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033464:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06c00/0x268e1a602cc2114 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9cab expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033470:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07c00/0x268e1a602cc20f1 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9c50 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033477:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07400/0x268e1a602cc20ce lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9bf5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033483:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06800/0x268e1a602cc20ab lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9b9a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033489:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859400/0x268e1a602cc2088 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9b3f expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033496:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08ea400/0x268e1a602cc2065 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9ae4 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033502:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eae00/0x268e1a602cc2042 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9a89 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033509:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0aa00/0x268e1a602cc201f lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9a2e expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033517:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3a600/0x268e1a602cc1ffc lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d99d3 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033524:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897800/0x268e1a602cc1fd9 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9978 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033530:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0b000/0x268e1a602cc1fb6 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d991d expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033537:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3be00/0x268e1a602cc1f93 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d98c2 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033543:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3a000/0x268e1a602cc1f70 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9867 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033550:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3b200/0x268e1a602cc1f4d lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d980c expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033556:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3b000/0x268e1a602cc1f2a lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d97b1 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033563:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3ba00/0x268e1a602cc1f07 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9756 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033569:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eaa00/0x268e1a602cc1ee4 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d96fb expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033576:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eb600/0x268e1a602cc1ec1 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d96a0 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033594:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08ea000/0x268e1a602cc1e9e lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9645 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033601:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eac00/0x268e1a602cc1e7b lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d95ea expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033607:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08ebc00/0x268e1a602cc1e58 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d958f expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033614:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eb400/0x268e1a602cc1e35 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9534 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033620:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4200/0x268e1a602cc1e12 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d94d9 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033627:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4800/0x268e1a602cc1def lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d947e expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033633:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5400/0x268e1a602cc1dcc lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9423 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033640:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4000/0x268e1a602cc1da9 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d93c8 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033646:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5200/0x268e1a602cc1d86 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d936d expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033652:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b7a00/0x268e1a602cc1d63 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9312 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033661:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0622c00/0x268e1a602cc1d40 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d92b7 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033668:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0623400/0x268e1a602cc1d1d lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d925c expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033674:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0623a00/0x268e1a602cc1cfa lrc: 1/0,0 mode: --/PR res: [0x240001b71:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9201 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033699:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0622800/0x268e1a602cc1cd7 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d91a6 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033706:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0623200/0x268e1a602cc1cb4 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d914b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033714:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0623e00/0x268e1a602cc1c91 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d90f0 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033722:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c367ba00/0x268e1a602cc1c6e lrc: 1/0,0 mode: --/PR res: [0x240001b71:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9095 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033730:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b7c00/0x268e1a602cc1c4b lrc: 1/0,0 mode: --/PR res: [0x240001b71:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d903a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033750:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b6600/0x268e1a602cc1c28 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8fdf expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033760:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b6800/0x268e1a602cc1c05 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8f84 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033769:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b6e00/0x268e1a602cc1be2 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8f29 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033775:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b6c00/0x268e1a602cc1bbf lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8ece expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033780:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b7800/0x268e1a602cc1b9c lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8e73 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033786:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b7200/0x268e1a602cc1b79 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8e18 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033791:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf8a00/0x268e1a602cc1b56 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8dbd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033796:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc1b25 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8d38 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033801:0:48572:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958600/0x268e1a602cc1b41 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d8d77 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827805.033806:0:48572:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 81 locks packed 00010000:00010000:2.0:1787827805.033832:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69c00/0x268e1a602cc25c1 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x51:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da8f9 expref: -99 pid: 48445 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827805.033854:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69400/0x268e1a602cc25ba lrc: 0/0,0 mode: --/CR res: [0x240001b71:0x50:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da8dd expref: -99 pid: 48444 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827805.033871:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d68200/0x268e1a602cc25ac lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x4f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0da8ac expref: -99 pid: 48444 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00080000:3.0:1787827805.033879:0:10954:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8ab3f0438000 x1874671280682624/t0(0) o103->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:17/18 lens 960/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1787827805.033914:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854200/0x268e1a602cc2597 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da866 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033967:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854600/0x268e1a602cc2574 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da80b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033986:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854e00/0x268e1a602cc2551 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da7b0 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.033997:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d69800/0x268e1a602cc252e lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da755 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034006:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0970200/0x268e1a602cc250b lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da6fa expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034013:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0970400/0x268e1a602cc24e8 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da69f expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034033:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0970600/0x268e1a602cc24c5 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da644 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034041:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0971e00/0x268e1a602cc24a2 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da5e9 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034054:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0970a00/0x268e1a602cc247f lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da58e expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034063:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0971c00/0x268e1a602cc245c lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da533 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034071:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2858000/0x268e1a602cc2439 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da4d8 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034080:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859a00/0x268e1a602cc2416 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da47d expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034088:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859000/0x268e1a602cc23f3 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da422 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034097:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859e00/0x268e1a602cc23d0 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da3c7 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034105:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859200/0x268e1a602cc23ad lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da36c expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034112:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2858a00/0x268e1a602cc238a lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da311 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034120:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a97800/0x268e1a602cc2367 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da2b6 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034141:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cea00/0x268e1a602cc2344 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da25b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034156:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf400/0x268e1a602cc2321 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da200 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034179:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf200/0x268e1a602cc22fe lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da1a5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034193:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cec00/0x268e1a602cc22db lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da14a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034202:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce200/0x268e1a602cc22b8 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da0ef expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034211:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07ce600/0x268e1a602cc2295 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da094 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034218:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cf800/0x268e1a602cc2272 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0da039 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034226:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f07cfc00/0x268e1a602cc224f lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9fde expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034234:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959a00/0x268e1a602cc222c lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9f83 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034247:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2400/0x268e1a602cc2209 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9f28 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034255:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f05b2600/0x268e1a602cc21e6 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9ecd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034263:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a600/0x268e1a602cc21c3 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9e72 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034276:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5800/0x268e1a602cc21a0 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9e17 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034289:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5e00/0x268e1a602cc217d lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9dbc expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034303:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06200/0x268e1a602cc215a lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9d61 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034317:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07600/0x268e1a602cc2137 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9d06 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034330:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06c00/0x268e1a602cc2114 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9cab expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034338:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07c00/0x268e1a602cc20f1 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9c50 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034351:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a07400/0x268e1a602cc20ce lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9bf5 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034359:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a06800/0x268e1a602cc20ab lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9b9a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034367:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3f2859400/0x268e1a602cc2088 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9b3f expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034383:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08ea400/0x268e1a602cc2065 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9ae4 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034396:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eae00/0x268e1a602cc2042 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9a89 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034410:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0aa00/0x268e1a602cc201f lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9a2e expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034424:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3a600/0x268e1a602cc1ffc lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d99d3 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034447:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897800/0x268e1a602cc1fd9 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9978 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034455:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0b000/0x268e1a602cc1fb6 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d991d expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034468:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3be00/0x268e1a602cc1f93 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d98c2 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034478:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3a000/0x268e1a602cc1f70 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9867 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034486:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3b200/0x268e1a602cc1f4d lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d980c expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034495:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3b000/0x268e1a602cc1f2a lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d97b1 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034520:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6d3ba00/0x268e1a602cc1f07 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9756 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034529:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eaa00/0x268e1a602cc1ee4 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d96fb expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034538:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eb600/0x268e1a602cc1ec1 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d96a0 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034547:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08ea000/0x268e1a602cc1e9e lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9645 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034556:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eac00/0x268e1a602cc1e7b lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d95ea expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034581:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08ebc00/0x268e1a602cc1e58 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d958f expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034591:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d08eb400/0x268e1a602cc1e35 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9534 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034600:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4200/0x268e1a602cc1e12 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d94d9 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034609:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4800/0x268e1a602cc1def lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d947e expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034617:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5400/0x268e1a602cc1dcc lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9423 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034626:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b4000/0x268e1a602cc1da9 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d93c8 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034635:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c81b5200/0x268e1a602cc1d86 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d936d expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034644:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b7a00/0x268e1a602cc1d63 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9312 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034653:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0622c00/0x268e1a602cc1d40 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d92b7 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034667:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0623400/0x268e1a602cc1d1d lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d925c expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034677:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0623a00/0x268e1a602cc1cfa lrc: 0/0,0 mode: --/PR res: [0x240001b71:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9201 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034686:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0622800/0x268e1a602cc1cd7 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d91a6 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034727:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0623200/0x268e1a602cc1cb4 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d914b expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034741:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0623e00/0x268e1a602cc1c91 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d90f0 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034756:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c367ba00/0x268e1a602cc1c6e lrc: 0/0,0 mode: --/PR res: [0x240001b71:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d9095 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034770:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b7c00/0x268e1a602cc1c4b lrc: 0/0,0 mode: --/PR res: [0x240001b71:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d903a expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034785:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b6600/0x268e1a602cc1c28 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8fdf expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034813:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b6800/0x268e1a602cc1c05 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8f84 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034827:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b6e00/0x268e1a602cc1be2 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8f29 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034837:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b6c00/0x268e1a602cc1bbf lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8ece expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034847:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b7800/0x268e1a602cc1b9c lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8e73 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034856:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d09b7200/0x268e1a602cc1b79 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8e18 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034865:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf8a00/0x268e1a602cc1b56 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8dbd expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034875:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1a00/0x268e1a602cc1b25 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8d38 expref: -99 pid: 48442 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827805.034884:0:48572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958600/0x268e1a602cc1b41 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d8d77 expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827805.034896:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827805.034898:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827805.034900:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827806.054107:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827807.078119:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827808.102063:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827808.102078:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827808.102084:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827808.102098:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2357 00000100:00080000:2.0:1787827808.102111:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827808.102122:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827808.102181:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280682752/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827808.102211:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827808.102217:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827808.102220:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827808.102227:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2357 00000100:00080000:2.0:1787827808.102233:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827808.102239:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827808.102251:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827808.102274:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280682752:192.168.203.124@tcp:250:kworker.0 00010000:00010000:1.0:1787827808.102274:0:27833:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 85/85 locks to regular blp list, 1 blwis in pool 00000100:00100000:2.0:1787827808.102274:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280682880/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827808.102303:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827808.102308:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827808.102310:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827808.102316:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2357 00000100:00080000:2.0:1787827808.102329:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827808.102335:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:1.0:1787827808.102337:0:48573:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 85 locks of 0 total in blp. (0 blwis in pool) 00000008:00100000:2.0:1787827808.102346:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00010000:00010000:1.0:1787827808.102356:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7c00/0x268e1a602cc1b10 lrc: 2/0,0 mode: PR/PR res: [0x240001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x772721ad1f0d8ceb expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1787827808.102371:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280683008/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827808.102374:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00010000:00010000:3.0:1787827808.102375:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827808.102414:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827808.102420:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827808.102422:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827808.102429:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2357 00000100:00080000:2.0:1787827808.102435:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827808.102445:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827808.102451:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00010000:00010000:1.0:1787827808.102453:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0be00/0x268e1a602cc103c lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x772721ad1f0d6ae0 expref: -99 pid: 48428 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1787827808.102459:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827808/real 1787827808] req@ffff8ab3f0438b40 x1874671280682752/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827808.102467:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280683136/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827808.102511:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854000/0x268e1a602cc1ae6 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d86e0 expref: -99 pid: 48431 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00080000:2.0:1787827808.102524:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827808.102533:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280682752/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827808.102538:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827808.102546:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827808.102559:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2357 00000080:00200000:1.0:1787827808.102562:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x52:0x0]: losing layout lock 00000100:00080000:0.0:1787827808.102570:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827808.102570:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827808.102582:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:1.0:1787827808.102589:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc1adf lrc: 2/0,0 mode: CR/CR res: [0x200001b72:0x51:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d86c4 expref: -99 pid: 48430 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000008:00100000:2.0:1787827808.102592:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:0.0:1787827808.102593:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827808.102608:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280682752:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827808.102615:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280683264/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000080:00200000:1.0:1787827808.102632:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x51:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.102650:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc1ad1 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x50:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x772721ad1f0d8693 expref: -99 pid: 48430 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1787827808.102657:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280682880:192.168.203.124@tcp:38:kworker.0 00010000:00010000:1.0:1787827808.102692:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc1abc lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d864d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000800:00000100:0.0:1787827808.102710:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000080:00200000:1.0:1787827808.102712:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x4f:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.102731:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959000/0x268e1a602cc1a99 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d85f2 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827808.102742:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827808/real 1787827808] req@ffff8ab3f043b0c0 x1874671280682880/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1787827808.102756:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x4e:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.102775:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959c00/0x268e1a602cc1a76 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8597 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827808.102781:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280682880/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827808.102819:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000080:00200000:1.0:1787827808.102820:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x4d:0x0]: losing layout lock 00000100:00080000:0.0:1787827808.102826:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827808.102834:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280682880:192.168.203.124@tcp:38:kworker.0 00010000:00010000:1.0:1787827808.102840:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958200/0x268e1a602cc1a53 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d853c expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827808.102846:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280683008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827808.102942:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000080:00200000:1.0:1787827808.102958:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x4c:0x0]: losing layout lock 00000100:00100000:0.0:1787827808.102963:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827808/real 1787827808] req@ffff8ab3f0438780 x1874671280683008/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1787827808.102980:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958a00/0x268e1a602cc1a30 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d84e1 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827808.102983:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280683008/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827808.103021:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827808.103026:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000080:00200000:1.0:1787827808.103029:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x4b:0x0]: losing layout lock 00000100:00100000:0.0:1787827808.103032:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280683008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827808.103047:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280683136:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827808.103049:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959800/0x268e1a602cc1a0d lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8486 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000800:00000100:0.0:1787827808.103073:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000080:00200000:1.0:1787827808.103074:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x4a:0x0]: losing layout lock 00000100:00100000:0.0:1787827808.103091:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827808/real 1787827808] req@ffff8ab3f043a580 x1874671280683136/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1787827808.103094:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958e00/0x268e1a602cc19ea lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d842b expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827808.103119:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280683136/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1787827808.103126:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x49:0x0]: losing layout lock 00000100:00080000:0.0:1787827808.103134:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827808.103141:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00010000:00010000:1.0:1787827808.103145:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0529400/0x268e1a602cc19c7 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d83d0 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827808.103146:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280683136:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827808.103162:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280683264:192.168.203.124@tcp:8:kworker.0 00000080:00200000:1.0:1787827808.103190:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x48:0x0]: losing layout lock 00000800:00000100:0.0:1787827808.103196:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00010000:00010000:1.0:1787827808.103215:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc19a4 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8375 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1787827808.103217:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827808/real 1787827808] req@ffff8ab3f043ad00 x1874671280683264/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827808.103262:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280683264/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000080:00200000:1.0:1787827808.103265:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x47:0x0]: losing layout lock 00000100:00080000:0.0:1787827808.103286:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00010000:00010000:1.0:1787827808.103290:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc1981 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d831a expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000100:00080000:0.0:1787827808.103291:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827808.103299:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280683264:192.168.203.124@tcp:8:kworker.0 00000080:00200000:1.0:1787827808.103332:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x46:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103346:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367b800/0x268e1a602cc195e lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d82bf expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103373:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x45:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103386:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367b200/0x268e1a602cc193b lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8264 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103402:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x44:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103413:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367ae00/0x268e1a602cc1918 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8209 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103429:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x43:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103439:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367a400/0x268e1a602cc18f5 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d81ae expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103457:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x42:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103468:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc18d2 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8153 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103484:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x41:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103495:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367a600/0x268e1a602cc18af lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d80f8 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103511:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x40:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103522:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367a200/0x268e1a602cc188c lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d809d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103539:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x3f:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103550:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367be00/0x268e1a602cc1869 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d8042 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103577:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x3e:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103588:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc1846 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7fe7 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103606:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x3d:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103616:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0529a00/0x268e1a602cc1823 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7f8c expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103633:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x3c:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103643:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0529800/0x268e1a602cc1800 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7f31 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103660:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x3b:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103671:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0528400/0x268e1a602cc17dd lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7ed6 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103688:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x3a:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103699:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0528a00/0x268e1a602cc17ba lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7e7b expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103715:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x39:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103726:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf9000/0x268e1a602cc1797 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7e20 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103743:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x38:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103754:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf8200/0x268e1a602cc1774 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7dc5 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103781:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x37:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103792:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0529c00/0x268e1a602cc1751 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7d6a expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103809:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x36:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103820:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0528600/0x268e1a602cc172e lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7d0f expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103837:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x35:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103847:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc170b lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7cb4 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103865:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x34:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103876:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc16e8 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7c59 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103951:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x33:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103965:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf8600/0x268e1a602cc16c5 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7bfe expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.103982:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x32:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.103994:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc16a2 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7ba3 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104011:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x31:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104021:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc167f lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7b48 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104047:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x30:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104058:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc165c lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7aed expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104075:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x2f:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104086:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031e00/0x268e1a602cc1639 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7a92 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104103:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x2e:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104113:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc1616 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7a37 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104130:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x2d:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104140:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc15f3 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d79dc expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104156:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x2c:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104166:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc15d0 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7981 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104182:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x2b:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104193:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc15ad lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7926 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104209:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x2a:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104220:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc158a lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d78cb expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104238:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x29:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104245:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc1567 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7870 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104255:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x28:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104261:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc1544 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7815 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104270:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x27:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104275:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc1521 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d77ba expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104284:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x26:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104289:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc14fe lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d775f expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104298:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x25:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104304:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7400/0x268e1a602cc14db lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7704 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104313:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x24:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104331:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0761e00/0x268e1a602cc14b8 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d76a9 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104341:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x23:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104347:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a200/0x268e1a602cc1495 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d764e expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104368:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x22:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104374:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0ac00/0x268e1a602cc1472 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d75f3 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104383:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x21:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104389:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf9c00/0x268e1a602cc144f lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7598 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104398:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x20:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104404:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf9200/0x268e1a602cc142c lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d753d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104413:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x1f:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104419:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf8800/0x268e1a602cc1409 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d74e2 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104428:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x1e:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104434:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf9600/0x268e1a602cc13e6 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7487 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104442:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x1d:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104448:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897000/0x268e1a602cc13c3 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d742c expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104456:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x1c:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104462:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0beb400/0x268e1a602cc13a0 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d73d1 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104476:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x1b:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104481:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897600/0x268e1a602cc137d lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7376 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104490:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x1a:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104496:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0896600/0x268e1a602cc135a lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d731b expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104504:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x19:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104510:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897200/0x268e1a602cc1337 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d72c0 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104518:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x18:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104524:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0896200/0x268e1a602cc1314 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7265 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104533:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x17:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104539:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897e00/0x268e1a602cc12f1 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d720a expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104547:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x16:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104553:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0beb600/0x268e1a602cc12ce lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d71af expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104561:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x15:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104567:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0761200/0x268e1a602cc12ab lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7154 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104581:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x14:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104587:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760e00/0x268e1a602cc1288 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d70f9 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104596:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x13:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104602:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760600/0x268e1a602cc1265 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d709e expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104610:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x12:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104616:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0761c00/0x268e1a602cc1242 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d7043 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104625:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x11:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104631:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760400/0x268e1a602cc121f lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d6fe8 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104639:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x10:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104645:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bea600/0x268e1a602cc11fc lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d6f8d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104655:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0xf:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104665:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bebc00/0x268e1a602cc11d9 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d6f32 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104681:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0xe:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104693:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0beb800/0x268e1a602cc11b6 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d6ed7 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104718:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0xd:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104729:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0beac00/0x268e1a602cc1193 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d6e7c expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104763:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0xc:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104775:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bea200/0x268e1a602cc1170 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d6e21 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104786:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0xb:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104792:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bea400/0x268e1a602cc114d lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d6dc6 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104821:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0xa:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104829:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc112a lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d6d6b expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104839:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x9:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104846:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc1107 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d6d10 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104857:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x8:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104864:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0ae00/0x268e1a602cc10e4 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d6cb5 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104874:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x7:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104881:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc10c1 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d6c5a expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104891:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x6:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104963:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6800/0x268e1a602cc109e lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d6bff expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104976:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x5:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104983:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6400/0x268e1a602cc107b lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d6ba4 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.104992:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x4:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.104999:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a400/0x268e1a602cc1051 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x772721ad1f0d6b2d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827808.105007:0:48573:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b72:0x3:0x0]: losing layout lock 00010000:00010000:1.0:1787827808.105013:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7a00/0x268e1a602cc1066 lrc: 2/0,0 mode: PR/PR res: [0x200001b72:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x772721ad1f0d6b5e expref: -99 pid: 48428 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827808.105024:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d200/0x268e1a602cc1012 lrc: 2/0,0 mode: PR/PR res: [0x200000002:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x772721ad1f0d6a46 expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105038:0:48573:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8ca00/0x268e1a602cc100b lrc: 2/0,0 mode: PR/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x772721ad1f0d6a2a expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827808.105069:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7c00/0x268e1a602cc1b10 lrc: 1/0,0 mode: --/PR res: [0x240001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d8ceb expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827808.105077:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0be00/0x268e1a602cc103c lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d6ae0 expref: -99 pid: 48428 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827808.105086:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854000/0x268e1a602cc1ae6 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d86e0 expref: -99 pid: 48431 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827808.105099:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc1adf lrc: 1/0,0 mode: --/CR res: [0x200001b72:0x51:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d86c4 expref: -99 pid: 48430 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827808.105107:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc1ad1 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x50:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d8693 expref: -99 pid: 48430 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827808.105118:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc1abc lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d864d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105130:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959000/0x268e1a602cc1a99 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d85f2 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105141:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959c00/0x268e1a602cc1a76 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8597 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105152:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958200/0x268e1a602cc1a53 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d853c expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105164:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958a00/0x268e1a602cc1a30 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d84e1 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105175:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959800/0x268e1a602cc1a0d lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8486 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105185:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958e00/0x268e1a602cc19ea lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d842b expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105198:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0529400/0x268e1a602cc19c7 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d83d0 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105211:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc19a4 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8375 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105223:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc1981 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d831a expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105234:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367b800/0x268e1a602cc195e lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d82bf expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105245:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367b200/0x268e1a602cc193b lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8264 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105255:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367ae00/0x268e1a602cc1918 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8209 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105266:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367a400/0x268e1a602cc18f5 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d81ae expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105277:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc18d2 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8153 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105289:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367a600/0x268e1a602cc18af lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d80f8 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105299:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367a200/0x268e1a602cc188c lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d809d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105311:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367be00/0x268e1a602cc1869 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8042 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105323:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc1846 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7fe7 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105336:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0529a00/0x268e1a602cc1823 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7f8c expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105348:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0529800/0x268e1a602cc1800 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7f31 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105360:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0528400/0x268e1a602cc17dd lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7ed6 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105368:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0528a00/0x268e1a602cc17ba lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7e7b expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105376:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf9000/0x268e1a602cc1797 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7e20 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105383:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf8200/0x268e1a602cc1774 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7dc5 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105390:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0529c00/0x268e1a602cc1751 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7d6a expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105396:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0528600/0x268e1a602cc172e lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7d0f expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105404:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc170b lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7cb4 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105413:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc16e8 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7c59 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105420:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf8600/0x268e1a602cc16c5 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7bfe expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105427:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc16a2 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7ba3 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105434:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc167f lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7b48 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105441:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc165c lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7aed expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105448:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031e00/0x268e1a602cc1639 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7a92 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105455:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc1616 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7a37 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105462:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc15f3 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d79dc expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105469:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc15d0 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7981 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105476:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc15ad lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7926 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105484:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc158a lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d78cb expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105491:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc1567 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7870 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105498:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc1544 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7815 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105505:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc1521 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d77ba expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105512:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc14fe lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d775f expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105519:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7400/0x268e1a602cc14db lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7704 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105526:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0761e00/0x268e1a602cc14b8 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d76a9 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105533:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a200/0x268e1a602cc1495 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d764e expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105540:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0ac00/0x268e1a602cc1472 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d75f3 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105547:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf9c00/0x268e1a602cc144f lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7598 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105555:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf9200/0x268e1a602cc142c lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d753d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105563:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf8800/0x268e1a602cc1409 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d74e2 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105570:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf9600/0x268e1a602cc13e6 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7487 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105577:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897000/0x268e1a602cc13c3 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d742c expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105584:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0beb400/0x268e1a602cc13a0 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d73d1 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105590:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897600/0x268e1a602cc137d lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7376 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105597:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0896600/0x268e1a602cc135a lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d731b expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105604:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897200/0x268e1a602cc1337 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d72c0 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105611:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0896200/0x268e1a602cc1314 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7265 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105618:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897e00/0x268e1a602cc12f1 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d720a expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105626:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0beb600/0x268e1a602cc12ce lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d71af expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105634:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0761200/0x268e1a602cc12ab lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7154 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105641:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760e00/0x268e1a602cc1288 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d70f9 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105648:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760600/0x268e1a602cc1265 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d709e expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105657:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0761c00/0x268e1a602cc1242 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7043 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105669:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760400/0x268e1a602cc121f lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6fe8 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105680:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bea600/0x268e1a602cc11fc lrc: 1/0,0 mode: --/PR res: [0x200001b72:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6f8d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105692:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bebc00/0x268e1a602cc11d9 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6f32 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105705:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0beb800/0x268e1a602cc11b6 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6ed7 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105718:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0beac00/0x268e1a602cc1193 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6e7c expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105728:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bea200/0x268e1a602cc1170 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6e21 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105736:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bea400/0x268e1a602cc114d lrc: 1/0,0 mode: --/PR res: [0x200001b72:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6dc6 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105743:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc112a lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6d6b expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105750:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc1107 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6d10 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105757:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0ae00/0x268e1a602cc10e4 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6cb5 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105764:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc10c1 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6c5a expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105771:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6800/0x268e1a602cc109e lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6bff expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105778:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6400/0x268e1a602cc107b lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6ba4 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105785:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a400/0x268e1a602cc1051 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6b2d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105791:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7a00/0x268e1a602cc1066 lrc: 1/0,0 mode: --/PR res: [0x200001b72:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d6b5e expref: -99 pid: 48428 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827808.105800:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d200/0x268e1a602cc1012 lrc: 1/0,0 mode: --/PR res: [0x200000002:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d6a46 expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.105807:0:48573:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8ca00/0x268e1a602cc100b lrc: 1/0,0 mode: --/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d6a2a expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827808.105813:0:48573:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 85 locks packed 00010000:00010000:1.0:1787827808.105842:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7c00/0x268e1a602cc1b10 lrc: 0/0,0 mode: --/PR res: [0x240001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d8ceb expref: -99 pid: 48442 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827808.105865:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0be00/0x268e1a602cc103c lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d6ae0 expref: -99 pid: 48428 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00080000:0.0:1787827808.105961:0:10955:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8ab3d22ffc00 x1874671280683392/t0(0) o103->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:17/18 lens 992/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1787827808.105961:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0854000/0x268e1a602cc1ae6 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d86e0 expref: -99 pid: 48431 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827808.106018:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1c00/0x268e1a602cc1adf lrc: 0/0,0 mode: --/CR res: [0x200001b72:0x51:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d86c4 expref: -99 pid: 48430 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827808.106034:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031c00/0x268e1a602cc1ad1 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x50:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d8693 expref: -99 pid: 48430 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827808.106048:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030000/0x268e1a602cc1abc lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d864d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106060:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959000/0x268e1a602cc1a99 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d85f2 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106079:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959c00/0x268e1a602cc1a76 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8597 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106090:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958200/0x268e1a602cc1a53 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d853c expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106101:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958a00/0x268e1a602cc1a30 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d84e1 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106112:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6959800/0x268e1a602cc1a0d lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8486 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106124:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6958e00/0x268e1a602cc19ea lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d842b expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106134:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0529400/0x268e1a602cc19c7 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d83d0 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106145:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166fe00/0x268e1a602cc19a4 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8375 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106208:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ec00/0x268e1a602cc1981 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d831a expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106220:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367b800/0x268e1a602cc195e lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d82bf expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106237:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367b200/0x268e1a602cc193b lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8264 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106248:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367ae00/0x268e1a602cc1918 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8209 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106259:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367a400/0x268e1a602cc18f5 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d81ae expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106270:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0000/0x268e1a602cc18d2 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8153 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106281:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367a600/0x268e1a602cc18af lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d80f8 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106293:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367a200/0x268e1a602cc188c lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d809d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106303:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c367be00/0x268e1a602cc1869 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d8042 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106314:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e400/0x268e1a602cc1846 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7fe7 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106325:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0529a00/0x268e1a602cc1823 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7f8c expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106341:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0529800/0x268e1a602cc1800 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7f31 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106352:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0528400/0x268e1a602cc17dd lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7ed6 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106365:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0528a00/0x268e1a602cc17ba lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7e7b expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106375:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf9000/0x268e1a602cc1797 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7e20 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106386:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf8200/0x268e1a602cc1774 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7dc5 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106397:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0529c00/0x268e1a602cc1751 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7d6a expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106409:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f0528600/0x268e1a602cc172e lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7d0f expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106420:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e000/0x268e1a602cc170b lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7cb4 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106430:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166ee00/0x268e1a602cc16e8 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7c59 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106445:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf8600/0x268e1a602cc16c5 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7bfe expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106456:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c166e200/0x268e1a602cc16a2 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7ba3 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106466:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1600/0x268e1a602cc167f lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7b48 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106476:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030400/0x268e1a602cc165c lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7aed expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106487:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031e00/0x268e1a602cc1639 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7a92 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106497:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0030600/0x268e1a602cc1616 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7a37 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106507:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c0031600/0x268e1a602cc15f3 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d79dc expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106518:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d600/0x268e1a602cc15d0 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7981 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106529:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0800/0x268e1a602cc15ad lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7926 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106545:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1800/0x268e1a602cc158a lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d78cb expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106556:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0400/0x268e1a602cc1567 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7870 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106566:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a0a00/0x268e1a602cc1544 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7815 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106576:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c70a1e00/0x268e1a602cc1521 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d77ba expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106586:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d045d000/0x268e1a602cc14fe lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d775f expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106596:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7400/0x268e1a602cc14db lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7704 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106607:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0761e00/0x268e1a602cc14b8 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d76a9 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106618:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a200/0x268e1a602cc1495 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d764e expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106629:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0ac00/0x268e1a602cc1472 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d75f3 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106644:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf9c00/0x268e1a602cc144f lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7598 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106658:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf9200/0x268e1a602cc142c lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d753d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106675:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf8800/0x268e1a602cc1409 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d74e2 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106695:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bf9600/0x268e1a602cc13e6 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7487 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106713:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897000/0x268e1a602cc13c3 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d742c expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106749:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0beb400/0x268e1a602cc13a0 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d73d1 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106768:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897600/0x268e1a602cc137d lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7376 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106781:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0896600/0x268e1a602cc135a lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d731b expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106793:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897200/0x268e1a602cc1337 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d72c0 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106810:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0896200/0x268e1a602cc1314 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7265 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106821:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0897e00/0x268e1a602cc12f1 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d720a expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106850:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0beb600/0x268e1a602cc12ce lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d71af expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106862:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0761200/0x268e1a602cc12ab lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7154 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106874:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760e00/0x268e1a602cc1288 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d70f9 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106886:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760600/0x268e1a602cc1265 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d709e expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106960:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0761c00/0x268e1a602cc1242 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d7043 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106973:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0760400/0x268e1a602cc121f lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6fe8 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.106984:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bea600/0x268e1a602cc11fc lrc: 0/0,0 mode: --/PR res: [0x200001b72:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6f8d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.107000:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bebc00/0x268e1a602cc11d9 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6f32 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.107011:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0beb800/0x268e1a602cc11b6 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6ed7 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.107021:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0beac00/0x268e1a602cc1193 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6e7c expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.107032:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bea200/0x268e1a602cc1170 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6e21 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.107042:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0bea400/0x268e1a602cc114d lrc: 0/0,0 mode: --/PR res: [0x200001b72:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6dc6 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.107052:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7200/0x268e1a602cc112a lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6d6b expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.107063:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7e00/0x268e1a602cc1107 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6d10 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.107074:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0ae00/0x268e1a602cc10e4 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6cb5 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.107085:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6000/0x268e1a602cc10c1 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6c5a expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.107101:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6800/0x268e1a602cc109e lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6bff expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.107112:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a6400/0x268e1a602cc107b lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6ba4 expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.107122:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3d0a0a400/0x268e1a602cc1051 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x772721ad1f0d6b2d expref: -99 pid: 48428 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.107133:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3f05a7a00/0x268e1a602cc1066 lrc: 0/0,0 mode: --/PR res: [0x200001b72:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d6b5e expref: -99 pid: 48428 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827808.107143:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8d200/0x268e1a602cc1012 lrc: 0/0,0 mode: --/PR res: [0x200000002:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d6a46 expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827808.107153:0:48573:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8ab3c584d000 lock: ffff8ab3c6a8ca00/0x268e1a602cc100b lrc: 0/0,0 mode: --/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x772721ad1f0d6a2a expref: -99 pid: 48254 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827808.107166:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827808.107168:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827808.107170:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827809.126163:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00020000:2.0:1787827809.638027:0:8:0:(osc_cache.c:999:osc_extent_wait()) extent ffff8ab3d05d6480@{[0 -> 0/1023], [2|0|+|cache|wiuY|ffff8ab3f050d6c0], [28672|1|+|-|ffff8ab3c70a0200|1024|0000000000000000]} lustre-OST0001-osc-ffff8ab3c584d000: wait ext to 0 timedout, recovery in progress? 00000008:00020000:2.0:1787827809.650958:0:8:0:(osc_cache.c:999:osc_extent_wait()) ### extent: 0000000038ce7ce5 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000a780220c/0x268e1a602cc1552 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x772721ad1f0d782a expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827810.149955:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827811.174131:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827812.197998:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827813.221973:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827813.221982:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827813.221985:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827813.221993:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2362 00000100:00080000:2.0:1787827813.222002:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827813.222008:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787827813.222014:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827813.222054:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280683520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827813.222075:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827813.222077:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827813.222079:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827813.222081:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2362 00000100:00080000:2.0:1787827813.222084:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827813.222086:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827813.222094:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827813.222106:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280683520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827813.222107:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280683648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827813.222131:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827813.222135:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827813.222137:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827813.222142:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2362 00000100:00080000:2.0:1787827813.222147:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827813.222151:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827813.222157:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827813.222171:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280683776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827813.222181:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827813.222188:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827813.222191:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827813.222193:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827813.222197:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2362 00000100:00080000:2.0:1787827813.222202:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827813.222206:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827813.222211:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827813.222223:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280683904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827813.222229:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827813/real 1787827813] req@ffff8ab3f0438f00 x1874671280683520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827813.222231:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827813.222237:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827813.222240:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827813.222256:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2362 00000100:00100000:0.0:1787827813.222264:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280683520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827813.222268:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827813.222275:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:0.0:1787827813.222278:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787827813.222279:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:0.0:1787827813.222284:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827813.222287:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280683520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787827813.222292:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280684032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827813.222310:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280683648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827813.222337:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827813.222346:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827813/real 1787827813] req@ffff8ab3f043ad00 x1874671280683648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827813.222354:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280683648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827813.222364:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827813.222366:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827813.222368:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280683648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827813.222372:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280683776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827813.222382:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827813.222387:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827813/real 1787827813] req@ffff8ab3f043a580 x1874671280683776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827813.222394:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280683776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827813.222399:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827813.222401:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827813.222403:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280683776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827813.222407:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280683904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827813.222416:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827813.222422:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827813/real 1787827813] req@ffff8ab3f0438780 x1874671280683904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827813.222429:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280683904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827813.222435:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827813.222436:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827813.222439:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280683904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827813.222442:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280684032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827813.222451:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827813.222456:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827813/real 1787827813] req@ffff8ab3f043b0c0 x1874671280684032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827813.222463:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280684032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827813.222468:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827813.222470:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827813.222472:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280684032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827814.246123:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827815.270079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827816.294054:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827817.318086:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827818.342060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827818.342068:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827818.342081:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827818.342098:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827818.342114:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2367 00000100:00080000:0.0:1787827818.342128:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827818.342138:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827818.342222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280684160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827818.342271:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827818.342276:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827818.342279:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827818.342285:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2367 00000100:00080000:0.0:1787827818.342290:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827818.342297:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827818.342310:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827818.342345:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280684288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827818.342369:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827818.342374:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827818.342377:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827818.342381:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2367 00000100:00080000:0.0:1787827818.342385:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827818.342398:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827818.342404:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827818.342412:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280684160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827818.342423:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280684416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827818.342438:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827818.342441:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827818.342443:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827818.342447:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2367 00000100:00080000:0.0:1787827818.342450:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827818.342455:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827818.342460:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827818.342471:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280684544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827818.342480:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827818.342485:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827818.342487:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827818.342491:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2367 00000100:00080000:0.0:1787827818.342497:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827818.342501:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827818.342506:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827818.342518:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280684672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827818.342570:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280684288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827818.342647:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280684416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827818.342672:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280684544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827818.342698:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280684672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827819.366134:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827820.389992:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827821.414113:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787827821.414303:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:3.0:1787827821.414357:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787827821.414375:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787827821.414424:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787827821.414434:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787827821.414447:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787827821.414460:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:0.0:1787827821.414485:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827818/real 1787827821] req@ffff8ab3f075e580 x1874671280684416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827821.414520:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280684416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827821.414535:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827821.414541:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827821.414547:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280684416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827821.414556:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827818/real 1787827821] req@ffff8ab3f075ed00 x1874671280684544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827821.414571:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280684544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827821.414583:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827821.414587:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827821.414591:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280684544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827821.414599:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827818/real 1787827821] req@ffff8ab3f075f480 x1874671280684672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827821.414613:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280684672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827821.414625:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827821.414631:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827821.414637:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280684672:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827821.414650:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827818/real 1787827821] req@ffff8ab3f075da40 x1874671280684160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827821.414674:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280684160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827821.414690:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827821.414696:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827821.414703:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280684160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827821.414716:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827818/real 1787827821] req@ffff8ab3f075cb40 x1874671280684288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827821.414741:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280684288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827821.414757:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827821.414764:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827821.414770:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280684288:192.168.203.124@tcp:38:kworker.0 00010000:00010000:1.0:1787827822.438045:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827823.462022:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827823.462036:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827823.462042:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827823.462057:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2373 00000100:00080000:0.0:1787827823.462070:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827823.462081:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787827823.462105:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827823.462133:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280684800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827823.462175:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827823.462180:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827823.462182:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827823.462187:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2373 00000100:00080000:0.0:1787827823.462192:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827823.462199:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827823.462212:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827823.462230:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280684928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827823.462240:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280684800:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827823.462251:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827823.462257:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827823.462260:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827823.462266:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2373 00000100:00080000:0.0:1787827823.462272:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827823.462276:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827823.462285:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827823.462303:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280685056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827823.462316:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827823.462321:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827823.462323:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827823.462327:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2373 00000100:00080000:0.0:1787827823.462332:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827823.462341:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:3.0:1787827823.462344:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787827823.462349:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827823.462366:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280685184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827823.462379:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827823.462384:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827823.462386:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827823.462391:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2373 00000100:00080000:0.0:1787827823.462396:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827823.462401:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827823.462407:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827823.462420:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280685312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827823.462454:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827823/real 1787827823] req@ffff8ab3f075cb40 x1874671280684800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827823.462493:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280684800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827823.462526:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827823.462536:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827823.462544:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280684800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827823.462583:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280684928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827823.462635:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827823.462664:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827823/real 1787827823] req@ffff8ab3f075da40 x1874671280684928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827823.462691:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280684928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827823.462707:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827823.462714:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827823.462721:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280684928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827823.462733:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280685056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827823.462829:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827823.462855:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827823/real 1787827823] req@ffff8ab3f075f480 x1874671280685056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827823.462881:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280685056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827823.462945:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827823.462954:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827823.462961:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280685056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827823.462974:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280685184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827823.463019:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827823.463043:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827823/real 1787827823] req@ffff8ab3f075ed00 x1874671280685184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827823.463078:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280685184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827823.463096:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827823.463102:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827823.463108:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280685184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827823.463119:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280685312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827823.463157:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827823.463180:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827823/real 1787827823] req@ffff8ab3f075e580 x1874671280685312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827823.463205:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280685312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827823.463226:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827823.463232:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827823.463238:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280685312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827824.486016:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827825.510075:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827826.534005:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827827.558123:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827828.582054:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827828.582066:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1787827828.582070:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827828.582072:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827828.582088:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2378 00000100:00080000:2.0:1787827828.582100:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827828.582110:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827828.582157:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280685440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827828.582194:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827828.582199:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827828.582201:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827828.582205:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2378 00000100:00080000:2.0:1787827828.582210:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827828.582215:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827828.582228:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827828.582243:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280685568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827828.582258:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827828.582267:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787827828.582270:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280685440:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827828.582271:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827828.582288:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2378 00000100:00080000:2.0:1787827828.582292:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827828.582297:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827828.582307:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827828.582323:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280685696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827828.582334:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827828.582339:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827828.582342:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827828.582348:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2378 00000100:00080000:2.0:1787827828.582364:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827828.582369:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827828.582374:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:3.0:1787827828.582384:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787827828.582392:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280685824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827828.582407:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827828.582412:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827828.582415:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827828.582423:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2378 00000100:00080000:2.0:1787827828.582427:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827828.582431:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827828.582438:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827828.582448:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280685952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827828.582481:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827828/real 1787827828] req@ffff8ab3f0438b40 x1874671280685440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827828.582505:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280685440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827828.582517:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827828.582522:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827828.582526:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280685440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827828.582563:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280685568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827828.582588:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827828.582600:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827828/real 1787827828] req@ffff8ab3f0439680 x1874671280685568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827828.582610:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280685568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827828.582616:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827828.582619:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827828.582622:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280685568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827828.582627:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280685696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827828.582640:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827828.582646:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827828/real 1787827828] req@ffff8ab3f043bc00 x1874671280685696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827828.582655:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280685696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827828.582661:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827828.582663:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827828.582666:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280685696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827828.582671:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280685824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827828.582684:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827828.582690:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827828/real 1787827828] req@ffff8ab3f04383c0 x1874671280685824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827828.582698:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280685824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827828.582706:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827828.582708:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827828.582710:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280685824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827828.582715:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280685952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827828.582726:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827828.582733:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827828/real 1787827828] req@ffff8ab3f04392c0 x1874671280685952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827828.582741:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280685952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827828.582748:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827828.582750:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827828.582752:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280685952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827829.606093:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827830.629986:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827831.654103:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827832.678057:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827833.701973:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827833.701982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827833.701986:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827833.701995:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2383 00000100:00080000:2.0:1787827833.702003:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827833.702010:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787827833.702045:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827833.702056:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280686080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827833.702077:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827833.702079:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827833.702081:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827833.702083:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2383 00000100:00080000:2.0:1787827833.702086:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827833.702089:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827833.702097:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827833.702116:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280686208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827833.702122:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280686080:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827833.702126:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827833.702131:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827833.702133:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827833.702136:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2383 00000100:00080000:2.0:1787827833.702138:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827833.702141:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827833.702146:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827833.702159:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280686336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827833.702173:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827833.702176:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827833.702179:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827833.702183:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2383 00000100:00080000:2.0:1787827833.702187:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827833.702192:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787827833.702196:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827833.702204:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827833.702217:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280686464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827833.702226:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827833.702229:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827833.702231:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827833.702236:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2383 00000100:00080000:2.0:1787827833.702240:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827833.702246:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827833.702246:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827833/real 1787827833] req@ffff8ab3f0439a40 x1874671280686080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827833.702255:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827833.702272:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280686592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827833.702275:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280686080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827833.702304:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827833.702308:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827833.702311:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280686080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827833.702343:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280686208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827833.702367:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827833.702381:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827833/real 1787827833] req@ffff8ab3f043a1c0 x1874671280686208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827833.702394:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280686208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827833.702400:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827833.702402:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827833.702405:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280686208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827833.702409:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280686336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827833.702422:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827833.702429:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827833/real 1787827833] req@ffff8ab3f043a940 x1874671280686336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827833.702437:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280686336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827833.702442:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827833.702444:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827833.702446:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280686336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827833.702449:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280686464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827833.702459:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827833.702464:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827833/real 1787827833] req@ffff8ab3f043b480 x1874671280686464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827833.702472:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280686464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827833.702478:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827833.702480:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827833.702481:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280686464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827833.702485:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280686592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827833.702493:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827833.702499:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827833/real 1787827833] req@ffff8ab3f0439e00 x1874671280686592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827833.702505:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280686592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827833.702516:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827833.702518:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827833.702520:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280686592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827834.726004:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827835.750036:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827836.774041:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827837.798033:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827838.822063:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827838.822076:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827838.822083:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827838.822098:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2388 00000100:00080000:0.0:1787827838.822127:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1787827838.822133:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827838.822139:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827838.822224:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280686720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827838.822275:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827838.822280:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827838.822290:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827838.822296:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2388 00000100:00080000:0.0:1787827838.822302:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827838.822307:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827838.822320:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827838.822363:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280686848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827838.822377:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827838.822384:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787827838.822384:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280686720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827838.822388:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827838.822394:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2388 00000100:00080000:0.0:1787827838.822398:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827838.822406:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827838.822419:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827838.822434:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280686976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827838.822453:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827838.822458:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827838.822460:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827838.822464:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2388 00000100:00080000:0.0:1787827838.822494:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827838.822500:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:3.0:1787827838.822509:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787827838.822512:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827838.822533:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280687104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827838.822545:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827838.822550:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827838.822553:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827838.822557:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2388 00000100:00080000:0.0:1787827838.822562:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827838.822566:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827838.822573:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827838.822592:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827838/real 1787827838] req@ffff8ab3f075cf00 x1874671280686720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827838.822593:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280687232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827838.822667:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280686720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827838.822726:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827838.822735:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827838.822742:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280686720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827838.822772:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280686848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827838.822803:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827838.822819:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827838/real 1787827838] req@ffff8ab3f075f0c0 x1874671280686848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827838.822833:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280686848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827838.822843:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827838.822847:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827838.822851:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280686848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827838.822858:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280686976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827838.822876:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827838.822929:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827838/real 1787827838] req@ffff8ab3f075de00 x1874671280686976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827838.822945:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280686976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827838.822954:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827838.822958:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827838.822962:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280686976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827838.822969:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280687104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827838.822990:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827838.823002:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827838/real 1787827838] req@ffff8ab3f075f840 x1874671280687104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827838.823014:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280687104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827838.823051:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827838.823055:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827838.823059:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280687104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827838.823065:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280687232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827838.823097:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827838.823112:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827838/real 1787827838] req@ffff8ab3f075e940 x1874671280687232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827838.823125:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280687232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827838.823135:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827838.823138:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827838.823142:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280687232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827839.846119:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827840.869997:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827841.894085:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827842.918005:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827843.942012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827843.942022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827843.942027:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827843.942037:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2393 00000100:00080000:0.0:1787827843.942054:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827843.942063:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787827843.942079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827843.942117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280687360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827843.942141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827843.942144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827843.942146:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827843.942149:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2393 00000100:00080000:0.0:1787827843.942156:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827843.942160:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827843.942169:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827843.942172:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280687360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827843.942183:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280687488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827843.942193:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827843.942195:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827843.942197:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827843.942200:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2393 00000100:00080000:0.0:1787827843.942202:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827843.942205:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827843.942208:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827843.942218:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280687616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827843.942224:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827843.942227:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827843.942229:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827843.942232:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2393 00000100:00080000:0.0:1787827843.942234:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827843.942237:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827843.942240:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:3.0:1787827843.942243:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827843.942252:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280687744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827843.942260:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827843.942263:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827843.942265:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827843.942268:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2393 00000100:00080000:0.0:1787827843.942270:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827843.942272:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827843.942275:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827843.942283:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280687872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827843.942299:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827843/real 1787827843] req@ffff8ab3f075c000 x1874671280687360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827843.942314:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280687360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827843.942323:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827843.942326:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827843.942329:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280687360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827843.942350:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280687488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827843.942367:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827843.942376:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827843/real 1787827843] req@ffff8ab3f075e940 x1874671280687488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827843.942384:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280687488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827843.942389:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827843.942391:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827843.942393:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280687488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827843.942397:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280687616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827843.942407:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827843.942413:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827843/real 1787827843] req@ffff8ab3f075f840 x1874671280687616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827843.942421:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280687616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827843.942426:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827843.942428:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827843.942430:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280687616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827843.942434:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280687744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827843.942444:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827843.942449:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827843/real 1787827843] req@ffff8ab3f075de00 x1874671280687744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827843.942457:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280687744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827843.942464:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827843.942466:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827843.942469:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280687744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827843.942472:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280687872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827843.942481:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827843.942486:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827843/real 1787827843] req@ffff8ab3f075f0c0 x1874671280687872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827843.942493:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280687872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827843.942500:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827843.942501:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827843.942503:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280687872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827844.966003:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827845.990049:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827847.014122:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827848.038124:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827849.061993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1787827849.062001:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827849.062003:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827849.062010:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827849.062022:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2398 00000100:00080000:3.0:1787827849.062035:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827849.062044:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827849.062087:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280688000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827849.062118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827849.062122:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827849.062124:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827849.062128:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2398 00000100:00080000:3.0:1787827849.062134:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827849.062141:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827849.062152:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827849.062156:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280688000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827849.062174:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280688128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827849.062191:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827849.062195:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827849.062197:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827849.062207:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2398 00000100:00080000:3.0:1787827849.062213:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787827849.062218:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827849.062222:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827849.062230:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827849.062244:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280688256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827849.062253:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827849.062256:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827849.062258:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827849.062263:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2398 00000100:00080000:3.0:1787827849.062268:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827849.062271:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827849/real 1787827849] req@ffff8ab3d0a403c0 x1874671280688000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827849.062275:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827849.062283:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827849.062298:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280688000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827849.062303:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280688384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827849.062347:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827849.062350:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827849.062353:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827849.062358:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827849.062362:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787827849.062367:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2398 00000100:00080000:3.0:1787827849.062374:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827849.062377:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280688000:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827849.062382:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827849.062393:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827849.062407:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280688512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827849.062420:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280688128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827849.062447:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827849.062460:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827849/real 1787827849] req@ffff8ab3d0a41e00 x1874671280688128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827849.062471:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280688128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827849.062478:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827849.062481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827849.062483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280688128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827849.062489:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280688256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827849.062509:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827849.062517:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827849/real 1787827849] req@ffff8ab3d0a412c0 x1874671280688256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827849.062526:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280688256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827849.062533:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827849.062536:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827849.062539:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280688256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827849.062544:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280688384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827849.062557:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827849.062563:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827849/real 1787827849] req@ffff8ab3d0a421c0 x1874671280688384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827849.062572:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280688384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827849.062580:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827849.062582:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827849.062584:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280688384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827849.062597:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280688512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827849.062610:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827849.062643:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827849/real 1787827849] req@ffff8ab3d0a43c00 x1874671280688512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827849.062653:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280688512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827849.062661:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827849.062664:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827849.062666:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280688512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827850.086123:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827851.110059:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827852.134093:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827853.158077:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827854.181998:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827854.182011:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827854.182016:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1787827854.182019:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827854.182028:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2403 00000100:00080000:2.0:1787827854.182042:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827854.182051:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827854.182100:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280688640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827854.182127:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827854.182130:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827854.182132:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827854.182135:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2403 00000100:00080000:2.0:1787827854.182138:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827854.182141:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827854.182150:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827854.182161:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280688768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827854.182169:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827854.182172:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827854.182174:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827854.182178:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2403 00000100:00080000:2.0:1787827854.182180:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827854.182183:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827854.182189:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280688640:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787827854.182191:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827854.182210:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280688896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827854.182223:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827854.182228:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827854.182231:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827854.182236:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2403 00000100:00080000:2.0:1787827854.182242:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827854.182248:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787827854.182253:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787827854.182259:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827854.182276:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280689024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827854.182284:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827854.182287:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827854.182289:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827854.182294:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2403 00000100:00080000:2.0:1787827854.182298:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827854.182303:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827854.182308:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827854/real 1787827854] req@ffff8ab3f043b840 x1874671280688640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827854.182309:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787827854.182336:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280689152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827854.182361:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280688640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827854.182378:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827854.182384:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827854.182388:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280688640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827854.182394:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280688768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827854.182417:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827854.182438:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827854/real 1787827854] req@ffff8ab3f0439e00 x1874671280688768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827854.182448:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280688768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827854.182455:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827854.182458:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827854.182460:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280688768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827854.182486:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280688896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827854.182507:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827854.182520:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827854/real 1787827854] req@ffff8ab3f043b480 x1874671280688896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827854.182544:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280688896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827854.182560:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827854.182563:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827854.182567:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280688896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827854.182573:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280689024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827854.182634:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827854.182652:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827854/real 1787827854] req@ffff8ab3f043a940 x1874671280689024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827854.182667:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280689024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827854.182680:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827854.182685:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827854.182689:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280689024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827854.182697:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280689152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827854.182722:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827854.182737:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827854/real 1787827854] req@ffff8ab3f043a1c0 x1874671280689152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827854.182754:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280689152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827854.182767:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827854.182771:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827854.182776:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280689152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827855.206034:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827856.230081:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827857.254019:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827858.278096:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827859.302047:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827859.302048:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827859.302063:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827859.302069:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827859.302082:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2408 00000100:00080000:3.0:1787827859.302094:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827859.302104:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827859.302155:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280689280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827859.302194:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827859.302199:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827859.302202:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827859.302206:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2408 00000100:00080000:3.0:1787827859.302211:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827859.302215:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827859.302228:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827859.302244:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280689408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827859.302261:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827859.302268:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827859.302270:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827859.302278:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2408 00000100:00100000:0.0:1787827859.302280:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280689280:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827859.302282:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827859.302297:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827859.302304:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827859.302321:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280689536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827859.302335:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827859.302339:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827859.302341:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827859.302347:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2408 00000100:00080000:3.0:1787827859.302373:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827859.302379:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827859.302387:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787827859.302397:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827859.302411:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280689664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827859.302427:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827859.302431:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827859.302434:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827859.302438:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2408 00000100:00080000:3.0:1787827859.302443:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827859.302448:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827859.302455:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827859.302473:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280689792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827859.302487:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827859/real 1787827859] req@ffff8ab3d0a40000 x1874671280689280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827859.302521:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280689280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827859.302536:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827859.302541:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827859.302576:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280689280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827859.302609:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280689408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827859.302644:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827859.302675:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827859/real 1787827859] req@ffff8ab3d0a430c0 x1874671280689408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827859.302692:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280689408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827859.302704:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827859.302708:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827859.302712:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280689408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827859.302720:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280689536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827859.302742:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827859.302755:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827859/real 1787827859] req@ffff8ab3d0a43840 x1874671280689536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827859.302769:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280689536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827859.302779:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827859.302782:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827859.302786:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280689536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827859.302793:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280689664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827859.302824:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827859.302837:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827859/real 1787827859] req@ffff8ab3d0a43480 x1874671280689664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827859.302852:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280689664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827859.302864:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827859.302868:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827859.302871:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280689664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827859.302878:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280689792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827859.302942:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827859.302956:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827859/real 1787827859] req@ffff8ab3d0a43c00 x1874671280689792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827859.302971:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280689792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827859.302983:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827859.302987:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827859.302991:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280689792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827860.326063:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827861.350082:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827862.374051:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827863.398008:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827864.421959:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827864.421968:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827864.421972:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827864.421981:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2414 00000100:00080000:3.0:1787827864.421989:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827864.421996:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827864.422012:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827864.422040:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280689920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827864.422070:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827864.422074:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827864.422076:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827864.422081:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2414 00000100:00080000:3.0:1787827864.422085:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827864.422090:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827864.422100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827864.422112:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280690048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827864.422121:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280689920:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827864.422124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827864.422129:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827864.422130:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827864.422133:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2414 00000100:00080000:3.0:1787827864.422136:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827864.422139:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827864.422143:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827864.422152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280690176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827864.422159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827864.422162:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827864.422163:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827864.422166:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2414 00000100:00080000:3.0:1787827864.422169:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827864.422172:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827864.422175:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787827864.422180:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827864.422188:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280690304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827864.422196:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827864.422199:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827864.422200:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827864.422203:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2414 00000100:00080000:3.0:1787827864.422205:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827864.422208:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827864.422211:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827864.422222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280690432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827864.422228:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827864/real 1787827864] req@ffff8ab3d0a421c0 x1874671280689920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827864.422244:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280689920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827864.422252:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827864.422255:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827864.422258:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280689920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827864.422276:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280690048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827864.422300:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827864.422310:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827864/real 1787827864] req@ffff8ab3d0a412c0 x1874671280690048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827864.422318:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280690048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827864.422324:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827864.422326:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827864.422328:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280690048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827864.422333:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280690176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827864.422350:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827864.422356:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827864/real 1787827864] req@ffff8ab3d0a41e00 x1874671280690176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827864.422364:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280690176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827864.422369:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827864.422371:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827864.422374:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280690176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827864.422377:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280690304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827864.422387:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827864.422393:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827864/real 1787827864] req@ffff8ab3d0a403c0 x1874671280690304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827864.422400:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280690304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827864.422406:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827864.422408:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827864.422410:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280690304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827864.422414:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280690432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827864.422423:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827864.422429:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827864/real 1787827864] req@ffff8ab3d0a43c00 x1874671280690432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827864.422436:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280690432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827864.422442:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827864.422444:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827864.422446:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280690432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827865.445997:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827866.470077:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827867.494025:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827868.518028:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827869.542078:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827869.542090:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827869.542099:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827869.542118:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827869.542130:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2419 00000100:00080000:3.0:1787827869.542139:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827869.542145:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827869.542213:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280690560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827869.542266:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827869.542270:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827869.542272:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827869.542274:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2419 00000100:00080000:3.0:1787827869.542277:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827869.542280:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827869.542288:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827869.542298:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280690688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827869.542305:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280690560:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827869.542309:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827869.542312:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827869.542313:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827869.542343:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2419 00000100:00080000:3.0:1787827869.542347:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827869.542352:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827869.542358:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827869.542374:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280690816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827869.542385:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827869.542389:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827869.542391:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827869.542396:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2419 00000100:00080000:3.0:1787827869.542401:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827869.542406:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787827869.542414:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787827869.542418:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827869.542446:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280690944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827869.542458:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827869.542462:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827869.542464:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827869.542469:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2419 00000100:00080000:3.0:1787827869.542493:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827869.542499:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827869.542518:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827869.542538:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827869/real 1787827869] req@ffff8ab3d0a43480 x1874671280690560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827869.542541:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280691072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827869.542597:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280690560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827869.542608:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827869.542612:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827869.542616:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280690560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827869.542654:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280690688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827869.542674:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827869.542683:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827869/real 1787827869] req@ffff8ab3d0a43840 x1874671280690688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827869.542691:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280690688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827869.542702:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827869.542704:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827869.542706:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280690688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827869.542709:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280690816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827869.542719:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827869.542725:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827869/real 1787827869] req@ffff8ab3d0a430c0 x1874671280690816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827869.542732:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280690816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827869.542737:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827869.542739:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827869.542741:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280690816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827869.542744:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280690944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827869.542754:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827869.542768:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827869/real 1787827869] req@ffff8ab3d0a40000 x1874671280690944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827869.542775:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280690944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827869.542781:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827869.542782:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827869.542785:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280690944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827869.542788:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280691072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827869.542797:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827869.542803:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827869/real 1787827869] req@ffff8ab3d0a43c00 x1874671280691072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827869.542809:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280691072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827869.542815:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827869.542817:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827869.542818:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280691072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827870.566062:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827871.590000:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827872.614032:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827873.638016:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827874.662078:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827874.662146:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827874.662153:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827874.662157:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827874.662166:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2424 00000100:00080000:0.0:1787827874.662175:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827874.662181:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827874.662216:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280691200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827874.662241:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827874.662244:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827874.662246:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827874.662249:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2424 00000100:00080000:0.0:1787827874.662252:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827874.662257:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827874.662266:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827874.662277:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280691328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827874.662287:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827874.662291:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280691200:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827874.662293:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827874.662300:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827874.662308:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2424 00000100:00080000:0.0:1787827874.662313:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827874.662320:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827874.662327:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827874.662345:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280691456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827874.662358:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787827874.662365:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827874.662369:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827874.662371:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827874.662377:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2424 00000100:00080000:0.0:1787827874.662382:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827874.662386:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827874.662392:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827874.662410:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827874/real 1787827874] req@ffff8ab3f075cf00 x1874671280691200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827874.662411:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280691584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827874.662464:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827874.662469:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827874.662471:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827874.662479:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2424 00000100:00100000:3.0:1787827874.662479:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280691200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827874.662488:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827874.662497:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:3.0:1787827874.662502:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827874.662504:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:3.0:1787827874.662510:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827874.662515:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280691200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827874.662521:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280691712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827874.662544:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280691328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827874.662580:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827874.662599:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827874/real 1787827874] req@ffff8ab3f075e580 x1874671280691328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827874.662616:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280691328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827874.662629:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827874.662632:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827874.662635:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280691328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827874.662642:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280691456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827874.662669:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827874.662695:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827874/real 1787827874] req@ffff8ab3f075ed00 x1874671280691456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827874.662712:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280691456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827874.662720:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827874.662723:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827874.662726:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280691456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827874.662731:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280691584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827874.662751:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827874.662760:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827874/real 1787827874] req@ffff8ab3f075f480 x1874671280691584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827874.662789:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280691584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827874.662802:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827874.662806:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827874.662810:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280691584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827874.662819:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280691712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827874.662848:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827874.662861:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827874/real 1787827874] req@ffff8ab3f075da40 x1874671280691712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827874.662872:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280691712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827874.662880:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827874.662883:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827874.662908:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280691712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827875.686113:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827876.710111:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827877.734057:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827878.758203:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827879.781979:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827879.781991:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827879.781995:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827879.782007:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2429 00010000:00010000:1.0:1787827879.782013:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827879.782020:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827879.782029:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827879.782078:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280691840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827879.782104:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827879.782107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827879.782108:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827879.782111:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2429 00000100:00080000:0.0:1787827879.782114:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827879.782116:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827879.782126:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827879.782137:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280691968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827879.782146:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827879.782149:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827879.782150:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827879.782153:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2429 00000100:00080000:0.0:1787827879.782157:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827879.782159:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280691840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827879.782163:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827879.782171:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827879.782178:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280692096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827879.782185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827879.782188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827879.782190:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827879.782193:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2429 00000100:00080000:0.0:1787827879.782195:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827879.782198:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827879.782202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827879.782210:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280692224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827879.782217:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827879.782219:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827879.782221:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827879.782223:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2429 00000100:00080000:0.0:1787827879.782228:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827879.782231:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827879.782234:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:3.0:1787827879.782240:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827879.782250:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280692352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827879.782302:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827879/real 1787827879] req@ffff8ab3f075cb40 x1874671280691840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827879.782319:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280691840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827879.782334:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827879.782338:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827879.782341:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280691840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827879.782364:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280691968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827879.782380:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827879.782389:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827879/real 1787827879] req@ffff8ab3f075da40 x1874671280691968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827879.782398:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280691968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827879.782420:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827879.782424:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827879.782427:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280691968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827879.782434:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280692096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827879.782455:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827879.782467:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827879/real 1787827879] req@ffff8ab3f075f480 x1874671280692096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827879.782476:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280692096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827879.782482:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827879.782485:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827879.782487:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280692096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827879.782492:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280692224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827879.782507:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827879.782513:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827879/real 1787827879] req@ffff8ab3f075ed00 x1874671280692224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827879.782521:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280692224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827879.782528:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827879.782529:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827879.782531:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280692224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827879.782534:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280692352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827879.782544:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827879.782550:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827879/real 1787827879] req@ffff8ab3f075e580 x1874671280692352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827879.782558:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280692352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827879.782565:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827879.782566:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827879.782568:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280692352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827880.806066:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827881.830004:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827882.854049:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827883.878012:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827884.902020:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827884.902031:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827884.902037:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827884.902048:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2434 00000100:00080000:3.0:1787827884.902059:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827884.902067:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827884.902068:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827884.902112:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280692480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827884.902139:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827884.902143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827884.902144:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827884.902147:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2434 00000100:00080000:3.0:1787827884.902150:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827884.902153:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827884.902163:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827884.902173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280692608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827884.902179:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280692480:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827884.902184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827884.902188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827884.902190:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827884.902193:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2434 00000100:00080000:3.0:1787827884.902196:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827884.902198:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827884.902202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827884.902211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280692736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827884.902223:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827884.902226:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827884.902227:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827884.902230:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2434 00000100:00080000:3.0:1787827884.902242:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827884.902245:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827884.902248:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827884.902257:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280692864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827884.902263:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827884.902266:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827884.902267:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827884.902278:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2434 00000100:00080000:3.0:1787827884.902281:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827884.902285:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827884.902291:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827884.902308:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280692608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827884.902315:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280692992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827884.902337:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280692736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827884.902357:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280692864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827884.902390:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280692992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787827885.926070:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827886.950107:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827887.974204:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827887.974265:0:10942:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787827887.974318:0:10942:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827887.974336:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827887.974397:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827887.974414:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827887.974428:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787827887.974443:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:3.0:1787827887.974477:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827884/real 1787827887] req@ffff8ab3d0a41e00 x1874671280692608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827887.974536:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280692608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827887.974554:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827887.974560:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827887.974564:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280692608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827887.974572:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827884/real 1787827887] req@ffff8ab3d0a412c0 x1874671280692736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827887.974586:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280692736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827887.974594:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827887.974598:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827887.974602:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280692736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827887.974609:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827884/real 1787827887] req@ffff8ab3d0a421c0 x1874671280692864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827887.974623:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280692864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827887.974633:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827887.974637:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827887.974640:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280692864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827887.974647:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827884/real 1787827887] req@ffff8ab3d0a403c0 x1874671280692480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827887.974660:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280692480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827887.974669:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827887.974672:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827887.974676:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280692480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827887.974706:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827884/real 1787827887] req@ffff8ab3d0a43c00 x1874671280692992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827887.974719:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280692992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827887.974729:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827887.974733:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827887.974736:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280692992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827888.998168:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827890.021981:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827890.021991:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827890.021995:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827890.022007:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2439 00000100:00080000:2.0:1787827890.022018:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827890.022027:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787827890.022073:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280693120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787827890.022102:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827890.022109:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827890.022115:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827890.022117:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827890.022122:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2439 00000100:00080000:2.0:1787827890.022127:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827890.022133:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827890.022146:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827890.022162:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280693248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827890.022164:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280693120:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787827890.022178:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827890.022182:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827890.022184:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827890.022187:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2439 00000100:00080000:2.0:1787827890.022190:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827890.022194:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827890.022199:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827890.022212:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280693376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827890.022224:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827890.022227:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827890.022230:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827890.022234:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2439 00000100:00080000:2.0:1787827890.022248:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827890.022257:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787827890.022258:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787827890.022265:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787827890.022282:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280693504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827890.022293:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827890.022297:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827890.022299:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827890.022305:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2439 00000100:00080000:2.0:1787827890.022310:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827890.022315:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787827890.022321:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827890.022326:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827890/real 1787827890] req@ffff8ab3f0439a40 x1874671280693120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827890.022350:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280693632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827890.022394:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280693120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827890.022407:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827890.022412:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827890.022416:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280693120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827890.022447:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280693248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827890.022485:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827890.022500:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827890/real 1787827890] req@ffff8ab3f04392c0 x1874671280693248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827890.022514:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280693248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827890.022523:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827890.022526:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827890.022530:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280693248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827890.022536:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280693376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827890.022556:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827890.022569:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827890/real 1787827890] req@ffff8ab3f04383c0 x1874671280693376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827890.022581:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280693376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827890.022597:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827890.022601:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827890.022605:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280693376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827890.022611:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280693504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827890.022634:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827890.022646:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827890/real 1787827890] req@ffff8ab3f043bc00 x1874671280693504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827890.022658:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280693504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827890.022669:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827890.022673:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827890.022676:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280693504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827890.022683:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280693632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827890.022703:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827890.022715:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827890/real 1787827890] req@ffff8ab3f0439680 x1874671280693632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827890.022727:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280693632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827890.022738:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827890.022741:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827890.022745:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280693632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827891.046199:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827892.070062:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827893.094090:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827894.118138:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827895.142065:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827895.142099:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827895.142112:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827895.142116:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827895.142126:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2444 00000100:00080000:0.0:1787827895.142136:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827895.142155:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827895.142193:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280693760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827895.142221:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827895.142225:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827895.142227:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827895.142232:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2444 00000100:00080000:0.0:1787827895.142235:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827895.142239:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827895.142247:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827895.142259:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280693888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827895.142265:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280693760:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827895.142271:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827895.142274:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827895.142276:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827895.142280:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2444 00000100:00080000:0.0:1787827895.142282:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827895.142286:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827895.142289:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827895.142314:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280694016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827895.142322:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827895.142324:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827895.142326:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827895.142329:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2444 00000100:00080000:0.0:1787827895.142332:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827895.142335:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:3.0:1787827895.142340:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787827895.142342:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827895.142353:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280694144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827895.142360:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827895.142363:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827895.142364:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827895.142367:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2444 00000100:00080000:0.0:1787827895.142369:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827895.142372:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827895.142375:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827895.142383:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280694272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827895.142397:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827895/real 1787827895] req@ffff8ab3f075cf00 x1874671280693760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827895.142420:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280693760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827895.142432:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827895.142437:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827895.142441:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280693760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827895.142466:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280693888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827895.142485:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827895.142494:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827895/real 1787827895] req@ffff8ab3f075f0c0 x1874671280693888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827895.142502:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280693888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827895.142508:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827895.142510:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827895.142513:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280693888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827895.142516:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280694016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827895.142525:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827895.142531:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827895/real 1787827895] req@ffff8ab3f075de00 x1874671280694016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827895.142538:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280694016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827895.142543:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827895.142545:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827895.142546:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280694016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827895.142549:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280694144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827895.142559:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827895.142565:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827895/real 1787827895] req@ffff8ab3f075f840 x1874671280694144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827895.142577:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280694144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827895.142586:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827895.142589:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827895.142592:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280694144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827895.142598:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280694272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827895.142618:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827895.142628:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827895/real 1787827895] req@ffff8ab3f075e940 x1874671280694272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827895.142636:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280694272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827895.142648:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827895.142650:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827895.142652:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280694272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827896.166077:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827897.190023:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827898.214062:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827899.238072:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827900.262018:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827900.262029:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827900.262034:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827900.262048:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2449 00010000:00010000:2.0:1787827900.262057:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827900.262059:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827900.262068:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827900.262114:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280694400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827900.262142:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827900.262146:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827900.262148:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827900.262152:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2449 00000100:00080000:3.0:1787827900.262156:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827900.262160:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827900.262172:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827900.262185:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280694528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827900.262186:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280694400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827900.262196:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827900.262199:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827900.262201:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827900.262205:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2449 00000100:00080000:3.0:1787827900.262208:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827900.262212:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827900.262216:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827900.262225:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280694656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827900.262234:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827900.262237:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827900.262239:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827900.262243:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2449 00000100:00080000:3.0:1787827900.262248:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827900.262252:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827900.262257:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787827900.262260:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827900.262282:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280694784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827900.262291:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827900.262301:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827900.262303:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827900.262310:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2449 00000100:00080000:3.0:1787827900.262313:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827900.262317:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827900.262322:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827900.262332:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280694912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827900.262333:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827900/real 1787827900] req@ffff8ab3d0a43c00 x1874671280694400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827900.262353:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280694400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827900.262364:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827900.262369:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827900.262374:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280694400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827900.262400:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280694528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827900.262419:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827900.262429:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827900/real 1787827900] req@ffff8ab3d0a403c0 x1874671280694528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827900.262437:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280694528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827900.262443:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827900.262446:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827900.262448:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280694528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827900.262451:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280694656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827900.262461:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827900.262467:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827900/real 1787827900] req@ffff8ab3d0a421c0 x1874671280694656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827900.262479:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280694656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827900.262485:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827900.262486:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827900.262488:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280694656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827900.262492:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280694784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827900.262502:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827900.262508:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827900/real 1787827900] req@ffff8ab3d0a412c0 x1874671280694784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827900.262516:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280694784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827900.262522:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827900.262524:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827900.262526:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280694784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827900.262529:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280694912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827900.262538:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827900.262543:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827900/real 1787827900] req@ffff8ab3d0a41e00 x1874671280694912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827900.262550:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280694912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827900.262562:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827900.262564:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827900.262566:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280694912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827901.286018:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827902.310049:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827903.334059:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827904.358095:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827905.381947:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827905.381955:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827905.381959:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827905.381967:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2455 00000100:00080000:3.0:1787827905.381975:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827905.381981:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827905.381996:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827905.382013:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280695040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827905.382033:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827905.382038:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827905.382040:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827905.382043:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2455 00000100:00080000:3.0:1787827905.382047:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827905.382052:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827905.382062:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827905.382076:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280695168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827905.382086:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827905.382087:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280695040:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827905.382089:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827905.382091:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827905.382094:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2455 00000100:00080000:3.0:1787827905.382098:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827905.382102:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827905.382107:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827905.382117:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280695296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827905.382125:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827905.382128:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827905.382130:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827905.382149:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2455 00000100:00080000:3.0:1787827905.382161:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827905.382167:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787827905.382173:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787827905.382174:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827905.382187:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280695424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827905.382198:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827905.382202:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827905.382204:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827905.382208:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2455 00000100:00080000:3.0:1787827905.382212:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827905.382217:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827905.382223:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827905.382266:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827905/real 1787827905] req@ffff8ab3d0a40000 x1874671280695040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827905.382269:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280695552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827905.382286:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280695040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827905.382296:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827905.382300:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827905.382304:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280695040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827905.382327:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280695168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827905.382345:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827905.382354:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827905/real 1787827905] req@ffff8ab3d0a430c0 x1874671280695168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827905.382362:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280695168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827905.382368:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827905.382370:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827905.382372:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280695168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827905.382376:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280695296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827905.382387:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827905.382393:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827905/real 1787827905] req@ffff8ab3d0a43840 x1874671280695296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827905.382400:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280695296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827905.382405:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827905.382406:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827905.382408:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280695296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827905.382411:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280695424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827905.382421:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827905.382426:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827905/real 1787827905] req@ffff8ab3d0a43480 x1874671280695424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827905.382433:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280695424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827905.382439:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827905.382441:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827905.382443:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280695424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827905.382446:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280695552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827905.382455:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827905.382460:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827905/real 1787827905] req@ffff8ab3d0a41e00 x1874671280695552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827905.382468:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280695552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827905.382474:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827905.382475:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827905.382477:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280695552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827906.406019:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827907.430102:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827908.454060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827909.478072:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827910.502011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827910.502059:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827910.502066:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827910.502081:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2460 00000100:00080000:0.0:1787827910.502107:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827910.502115:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827910.502129:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827910.502151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280695680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827910.502175:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827910.502178:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827910.502180:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827910.502182:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2460 00000100:00080000:0.0:1787827910.502187:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827910.502203:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827910.502212:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827910.502225:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280695808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827910.502237:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280695680:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827910.502257:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827910.502260:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827910.502262:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827910.502266:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2460 00000100:00080000:0.0:1787827910.502269:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827910.502273:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827910.502277:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827910.502286:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280695936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827910.502294:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827910.502297:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827910.502298:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827910.502300:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2460 00000100:00080000:0.0:1787827910.502317:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827910.502320:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827910.502324:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:3.0:1787827910.502329:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827910.502335:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280696064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827910.502343:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827910.502346:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827910.502348:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827910.502352:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2460 00000100:00080000:0.0:1787827910.502357:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827910.502360:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827910.502364:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827910.502373:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280696192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827910.502401:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827910/real 1787827910] req@ffff8ab3f075c000 x1874671280695680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827910.502418:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280695680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827910.502428:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827910.502431:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827910.502435:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280695680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827910.502458:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280695808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827910.502476:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827910.502485:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827910/real 1787827910] req@ffff8ab3f075e940 x1874671280695808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827910.502493:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280695808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827910.502499:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827910.502501:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827910.502504:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280695808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827910.502507:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280695936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827910.502518:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827910.502523:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827910/real 1787827910] req@ffff8ab3f075f840 x1874671280695936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827910.502531:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280695936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827910.502537:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827910.502538:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827910.502540:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280695936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827910.502544:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280696064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827910.502555:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827910.502560:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827910/real 1787827910] req@ffff8ab3f075de00 x1874671280696064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827910.502573:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280696064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827910.502580:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827910.502587:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827910.502590:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280696064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827910.502594:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280696192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827910.502606:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827910.502612:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827910/real 1787827910] req@ffff8ab3f075f0c0 x1874671280696192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827910.502620:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280696192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827910.502628:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827910.502630:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827910.502632:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280696192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827911.526055:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827912.550054:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827913.574082:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827914.598061:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827915.622024:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827915.622035:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827915.622039:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827915.622050:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2465 00000100:00080000:3.0:1787827915.622061:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827915.622070:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827915.622076:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827915.622106:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280696320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827915.622132:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827915.622135:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827915.622136:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827915.622139:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2465 00000100:00080000:3.0:1787827915.622141:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827915.622144:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827915.622151:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827915.622176:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280696448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827915.622185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827915.622188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787827915.622191:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280696320:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827915.622191:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827915.622200:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2465 00000100:00080000:3.0:1787827915.622203:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827915.622206:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827915.622210:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827915.622220:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280696576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827915.622227:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827915.622230:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827915.622231:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827915.622235:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2465 00000100:00080000:3.0:1787827915.622249:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827915.622253:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827915.622257:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787827915.622262:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827915.622267:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280696704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827915.622274:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827915.622277:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827915.622278:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827915.622281:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2465 00000100:00080000:3.0:1787827915.622284:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827915.622287:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827915.622290:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827915.622298:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280696832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827915.622320:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827915/real 1787827915] req@ffff8ab3d0a412c0 x1874671280696320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827915.622337:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280696320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827915.622345:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827915.622348:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827915.622351:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280696320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827915.622370:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280696448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827915.622387:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827915.622395:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827915/real 1787827915] req@ffff8ab3d0a421c0 x1874671280696448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827915.622402:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280696448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827915.622408:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827915.622411:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827915.622413:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280696448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827915.622417:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280696576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827915.622427:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827915.622433:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827915/real 1787827915] req@ffff8ab3d0a403c0 x1874671280696576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827915.622439:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280696576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827915.622445:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827915.622447:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827915.622449:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280696576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827915.622452:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280696704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827915.622461:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827915.622467:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827915/real 1787827915] req@ffff8ab3d0a43c00 x1874671280696704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827915.622474:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280696704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827915.622480:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827915.622482:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827915.622483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280696704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827915.622487:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280696832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827915.622496:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827915.622502:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827915/real 1787827915] req@ffff8ab3d0a41e00 x1874671280696832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827915.622509:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280696832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827915.622515:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827915.622516:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827915.622518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280696832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827916.646043:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827917.670085:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827918.694032:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827919.718002:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827920.741977:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827920.741993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827920.741998:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827920.742009:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2470 00010000:00010000:2.0:1787827920.742013:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827920.742021:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827920.742030:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827920.742062:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280696960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827920.742085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827920.742088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827920.742089:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827920.742092:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2470 00000100:00080000:0.0:1787827920.742096:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827920.742099:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827920.742107:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827920.742114:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280696960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827920.742129:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280697088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827920.742140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827920.742143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827920.742144:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827920.742147:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2470 00000100:00080000:0.0:1787827920.742150:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827920.742153:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827920.742158:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827920.742168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280697216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827920.742175:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827920.742177:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827920.742179:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787827920.742184:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787827920.742187:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2470 00000100:00080000:0.0:1787827920.742191:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827920.742195:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827920.742199:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827920.742207:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280697344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827920.742213:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827920.742216:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827920.742218:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827920.742221:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2470 00000100:00080000:0.0:1787827920.742225:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827920.742229:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827920/real 1787827920] req@ffff8ab3f075cf00 x1874671280696960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827920.742232:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827920.742238:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827920.742251:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280697472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827920.742255:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280696960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827920.742277:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827920.742281:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827920.742286:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280696960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827920.742306:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280697088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827920.742322:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827920.742330:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827920/real 1787827920] req@ffff8ab3f075e580 x1874671280697088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827920.742339:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280697088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827920.742344:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827920.742346:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827920.742349:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280697088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827920.742352:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280697216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827920.742362:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827920.742368:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827920/real 1787827920] req@ffff8ab3f075ed00 x1874671280697216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827920.742375:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280697216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827920.742380:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827920.742382:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827920.742384:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280697216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827920.742387:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280697344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827920.742397:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827920.742403:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827920/real 1787827920] req@ffff8ab3f075f480 x1874671280697344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827920.742411:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280697344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827920.742417:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827920.742419:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827920.742421:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280697344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827920.742424:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280697472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827920.742440:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827920.742468:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827920/real 1787827920] req@ffff8ab3f075da40 x1874671280697472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827920.742483:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280697472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827920.742496:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827920.742501:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827920.742505:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280697472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827921.766016:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827922.790094:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827923.814189:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827924.838086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827925.862076:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827925.862135:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827925.862143:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827925.862147:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827925.862157:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2475 00000100:00080000:3.0:1787827925.862167:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827925.862173:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827925.862224:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280697600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827925.862258:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827925.862261:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827925.862263:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827925.862267:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2475 00000100:00080000:3.0:1787827925.862270:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827925.862273:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827925.862283:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827925.862295:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280697728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827925.862305:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827925.862309:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827925.862311:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827925.862315:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2475 00000100:00080000:3.0:1787827925.862317:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827925.862322:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827925.862326:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827925.862334:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280697856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827925.862363:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827925.862365:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827925.862367:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827925.862370:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2475 00000100:00080000:3.0:1787827925.862372:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827925.862375:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827925.862380:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827925.862393:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280697984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827925.862397:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280697600:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827925.862403:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827925.862415:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827925.862417:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827925.862422:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2475 00000100:00080000:3.0:1787827925.862424:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827925.862426:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827925.862429:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827925.862444:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280698112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827925.862528:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827925.862619:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827925/real 1787827925] req@ffff8ab3d0a43480 x1874671280697600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827925.862652:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280697600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827925.862672:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827925.862680:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827925.862686:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280697600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827925.862712:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280697728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827925.862740:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827925.862754:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827925/real 1787827925] req@ffff8ab3d0a43840 x1874671280697728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827925.862783:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280697728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827925.862792:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827925.862796:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827925.862800:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280697728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827925.862807:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280697856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827925.862826:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827925.862835:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827925/real 1787827925] req@ffff8ab3d0a430c0 x1874671280697856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827925.862846:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280697856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827925.862864:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827925.862867:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827925.862871:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280697856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827925.863005:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280697984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827925.863052:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827925.863070:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827925/real 1787827925] req@ffff8ab3d0a40000 x1874671280697984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827925.863113:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280697984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827925.863133:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827925.863137:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827925.863142:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280697984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827925.863150:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280698112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827925.863178:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827925.863192:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827925/real 1787827925] req@ffff8ab3d0a41e00 x1874671280698112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827925.863207:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280698112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827925.863220:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827925.863224:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827925.863228:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280698112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827926.886184:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827927.910119:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827928.934034:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827929.958082:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827930.982115:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827930.982119:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827930.982127:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827930.982131:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827930.982141:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2480 00000100:00080000:0.0:1787827930.982149:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827930.982155:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827930.982201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280698240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827930.982229:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827930.982231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827930.982233:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827930.982235:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2480 00000100:00080000:0.0:1787827930.982238:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827930.982240:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827930.982248:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827930.982260:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280698368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827930.982268:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827930.982270:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827930.982271:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827930.982274:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2480 00000100:00080000:0.0:1787827930.982276:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827930.982287:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827930.982290:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827930.982297:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280698496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827930.982309:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280698240:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827930.982311:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827930.982320:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827930.982326:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827930.982330:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2480 00000100:00080000:0.0:1787827930.982333:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827930.982335:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827930.982339:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827930.982351:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280698624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827930.982358:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827930.982360:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827930.982362:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827930.982365:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2480 00000100:00080000:0.0:1787827930.982369:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827930.982372:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827930.982375:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827930.982384:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280698752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827930.982410:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827930.982484:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827930/real 1787827930] req@ffff8ab3f075cb40 x1874671280698240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827930.982511:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280698240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827930.982529:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827930.982533:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827930.982538:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280698240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827930.982546:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280698368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827930.982574:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827930.982592:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827930/real 1787827930] req@ffff8ab3f075da40 x1874671280698368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827930.982610:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280698368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827930.982621:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827930.982626:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827930.982630:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280698368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827930.982659:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280698496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827930.982678:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827930.982688:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827930/real 1787827930] req@ffff8ab3f075f480 x1874671280698496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827930.982698:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280698496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827930.982706:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827930.982708:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827930.982711:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280698496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827930.982716:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280698624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827930.982729:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827930.982737:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827930/real 1787827930] req@ffff8ab3f075ed00 x1874671280698624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827930.982746:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280698624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827930.982755:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827930.982757:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827930.982759:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280698624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827930.982764:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280698752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827930.982776:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827930.982783:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827930/real 1787827930] req@ffff8ab3f075e580 x1874671280698752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827930.982792:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280698752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827930.982800:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827930.982802:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827930.982805:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280698752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827932.006066:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827933.030014:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827934.054134:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827935.078204:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827936.102079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827936.102113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827936.102126:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827936.102133:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827936.102149:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2485 00000100:00080000:3.0:1787827936.102165:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827936.102176:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827936.102237:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280698880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827936.102281:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827936.102286:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827936.102289:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827936.102293:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2485 00000100:00080000:3.0:1787827936.102297:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827936.102302:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827936.102317:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827936.102334:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280699008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827936.102349:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827936.102357:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280698880:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827936.102357:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827936.102364:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827936.102377:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2485 00000100:00080000:3.0:1787827936.102382:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827936.102386:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827936.102392:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827936.102409:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280699136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827936.102422:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827936.102426:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827936.102428:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827936.102435:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2485 00000100:00080000:3.0:1787827936.102455:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827936.102461:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827936.102469:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787827936.102474:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827936.102494:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280699264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827936.102509:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827936.102514:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827936.102516:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827936.102521:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2485 00000100:00080000:3.0:1787827936.102525:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827936.102530:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827936.102537:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827936.102551:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280699392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827936.102574:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827936/real 1787827936] req@ffff8ab3d0a43c00 x1874671280698880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827936.102604:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280698880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827936.102620:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827936.102625:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827936.102630:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280698880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827936.102663:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280699008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827936.102690:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827936.102704:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827936/real 1787827936] req@ffff8ab3d0a403c0 x1874671280699008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827936.102718:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280699008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827936.102728:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827936.102732:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827936.102736:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280699008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827936.102743:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280699136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827936.102762:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827936.102772:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827936/real 1787827936] req@ffff8ab3d0a421c0 x1874671280699136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827936.102785:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280699136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827936.102794:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827936.102797:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827936.102800:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280699136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827936.102807:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280699264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827936.102825:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827936.102834:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827936/real 1787827936] req@ffff8ab3d0a412c0 x1874671280699264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827936.102847:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280699264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827936.102858:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827936.102861:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827936.102865:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280699264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827936.102871:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280699392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827936.102922:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827936.102936:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827936/real 1787827936] req@ffff8ab3d0a41e00 x1874671280699392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827936.102950:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280699392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827936.102961:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827936.102965:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827936.102968:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280699392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827937.126030:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827938.150113:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827939.174089:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827940.198026:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827941.222004:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827941.222015:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827941.222021:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827941.222026:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827941.222036:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2490 00000100:00080000:0.0:1787827941.222046:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827941.222053:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827941.222091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280699520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827941.222116:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827941.222119:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827941.222120:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827941.222124:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2490 00000100:00080000:0.0:1787827941.222127:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827941.222131:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827941.222139:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827941.222144:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280699520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827941.222153:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280699648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827941.222163:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827941.222165:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827941.222167:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827941.222170:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2490 00000100:00080000:0.0:1787827941.222173:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827941.222175:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827941.222178:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827941.222186:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280699776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827941.222192:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827941.222195:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827941.222196:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827941.222201:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2490 00000800:00000100:3.0:1787827941.222201:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787827941.222207:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827941.222210:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827941.222215:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827941.222222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280699904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827941.222228:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827941.222231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827941.222233:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827941.222235:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2490 00000100:00080000:0.0:1787827941.222239:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827941.222241:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827941.222244:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827941.222261:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280700032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827941.222267:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827941/real 1787827941] req@ffff8ab3f075cf00 x1874671280699520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827941.222285:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280699520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827941.222294:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827941.222297:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827941.222301:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280699520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827941.222319:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280699648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827941.222334:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827941.222342:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827941/real 1787827941] req@ffff8ab3f075f0c0 x1874671280699648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827941.222350:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280699648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827941.222355:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827941.222357:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827941.222359:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280699648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827941.222363:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280699776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827941.222372:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827941.222377:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827941/real 1787827941] req@ffff8ab3f075de00 x1874671280699776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827941.222384:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280699776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827941.222389:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827941.222391:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827941.222392:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280699776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827941.222396:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280699904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827941.222405:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827941.222411:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827941/real 1787827941] req@ffff8ab3f075f840 x1874671280699904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827941.222418:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280699904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827941.222425:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827941.222426:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827941.222428:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280699904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827941.222431:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280700032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827941.222441:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827941.222446:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827941/real 1787827941] req@ffff8ab3f075e940 x1874671280700032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827941.222454:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280700032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827941.222470:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827941.222471:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827941.222473:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280700032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827942.246126:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827943.270166:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827944.294156:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827945.318066:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827946.342022:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827946.342036:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827946.342043:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827946.342056:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2495 00000100:00080000:0.0:1787827946.342069:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787827946.342072:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827946.342080:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827946.342129:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280700160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827946.342162:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827946.342166:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827946.342168:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827946.342172:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2495 00000100:00080000:0.0:1787827946.342176:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827946.342182:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827946.342194:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827946.342211:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280700288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827946.342220:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280700160:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827946.342228:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827946.342236:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827946.342239:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827946.342244:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2495 00000100:00080000:0.0:1787827946.342249:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827946.342254:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827946.342260:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827946.342273:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280700416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827946.342284:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827946.342288:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827946.342290:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827946.342297:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2495 00000800:00000100:3.0:1787827946.342297:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787827946.342305:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827946.342312:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827946.342318:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827946.342332:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280700544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827946.342342:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827946.342352:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827946/real 1787827946] req@ffff8ab3f075c000 x1874671280700160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827946.342357:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827946.342362:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827946.342371:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2495 00000100:00080000:0.0:1787827946.342376:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827946.342377:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280700160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827946.342386:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:3.0:1787827946.342393:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827946.342395:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:3.0:1787827946.342400:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827946.342405:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280700160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827946.342415:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280700672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827946.342430:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280700288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827946.342450:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827946.342461:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827946/real 1787827946] req@ffff8ab3f075e940 x1874671280700288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827946.342474:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280700288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827946.342483:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827946.342487:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827946.342490:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280700288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827946.342497:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280700416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827946.342517:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827946.342529:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827946/real 1787827946] req@ffff8ab3f075f840 x1874671280700416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827946.342541:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280700416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827946.342550:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827946.342553:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827946.342557:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280700416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827946.342563:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280700544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827946.342583:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827946.342593:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827946/real 1787827946] req@ffff8ab3f075de00 x1874671280700544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827946.342603:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280700544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827946.342613:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827946.342616:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827946.342620:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280700544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827946.342636:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280700672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827946.342654:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827946.342667:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827946/real 1787827946] req@ffff8ab3f075f0c0 x1874671280700672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827946.342689:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280700672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827946.342702:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827946.342705:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827946.342707:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280700672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827947.366038:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827948.390054:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827949.414056:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827950.438077:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827951.462010:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827951.462018:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827951.462022:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827951.462034:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2501 00000100:00080000:3.0:1787827951.462044:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787827951.462052:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827951.462053:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827951.462109:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280700800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827951.462132:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827951.462135:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827951.462136:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827951.462139:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2501 00000100:00080000:3.0:1787827951.462141:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827951.462145:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827951.462164:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827951.462173:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280700928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827951.462178:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280700800:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827951.462186:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827951.462189:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827951.462191:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827951.462194:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2501 00000100:00080000:3.0:1787827951.462196:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827951.462199:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827951.462202:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827951.462213:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280701056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827951.462219:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827951.462222:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827951.462223:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827951.462225:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2501 00000100:00080000:3.0:1787827951.462228:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827951.462230:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827951.462234:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827951.462241:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280701184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827951.462248:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827951.462250:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827951.462251:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827951.462254:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2501 00000100:00080000:3.0:1787827951.462256:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827951.462259:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827951.462262:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827951.462270:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280701312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827951.462290:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280700928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827951.462326:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280701056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827951.462350:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280701184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827951.462368:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280701312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787827952.486038:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827953.510124:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827954.534181:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787827954.534373:0:10944:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787827954.534449:0:10944:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827954.534491:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827954.534559:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827954.534576:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827954.534599:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787827954.534623:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:0.0:1787827954.534640:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827951/real 1787827954] req@ffff8ab3d0a430c0 x1874671280700928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827954.534706:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280700928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827954.534769:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827954.534781:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827954.534790:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280700928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827954.534806:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827951/real 1787827954] req@ffff8ab3d0a43840 x1874671280701056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827954.534834:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280701056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827954.534851:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827954.534860:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827954.534867:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280701056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827954.534881:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827951/real 1787827954] req@ffff8ab3d0a43480 x1874671280701184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827954.534988:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280701184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827954.535011:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827954.535018:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827954.535026:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280701184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827954.535041:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827951/real 1787827954] req@ffff8ab3d0a41e00 x1874671280701312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827954.535064:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280701312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827954.535085:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827954.535094:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827954.535103:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280701312:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827954.535117:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827951/real 1787827954] req@ffff8ab3d0a40000 x1874671280700800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827954.535145:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280700800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827954.535162:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827954.535170:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827954.535177:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280700800:192.168.203.124@tcp:250:kworker.0 00010000:00010000:2.0:1787827955.558123:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827956.582134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827956.582147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1787827956.582151:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827956.582154:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827956.582170:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2506 00000100:00080000:3.0:1787827956.582182:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827956.582192:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827956.582235:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280701440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827956.582266:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827956.582271:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827956.582273:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827956.582278:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2506 00000100:00080000:3.0:1787827956.582282:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827956.582288:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827956.582311:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827956.582316:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280701440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827956.582332:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280701568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827956.582348:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827956.582353:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827956.582356:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827956.582361:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2506 00000100:00080000:3.0:1787827956.582367:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827956.582371:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827956.582377:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827956.582391:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280701696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827956.582412:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827956.582416:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827956.582419:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787827956.582430:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827956.582432:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2506 00000100:00080000:3.0:1787827956.582442:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827956.582447:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827956.582457:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827956.582482:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280701824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827956.582501:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827956.582509:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827956.582513:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827956.582521:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827956/real 1787827956] req@ffff8ab3d0a412c0 x1874671280701440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827956.582529:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2506 00000100:00080000:3.0:1787827956.582541:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827956.582552:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827956.582569:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827956.582600:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280701440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827956.582600:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280701952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827956.582667:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827956.582678:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827956.582686:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280701440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827956.582732:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280701568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827956.582792:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827956.582818:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827956/real 1787827956] req@ffff8ab3d0a421c0 x1874671280701568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827956.582857:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280701568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827956.582939:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827956.582947:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827956.582954:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280701568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827956.582967:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280701696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827956.583013:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827956.583037:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827956/real 1787827956] req@ffff8ab3d0a403c0 x1874671280701696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827956.583057:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280701696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827956.583071:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827956.583078:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827956.583083:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280701696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827956.583094:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280701824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827956.583129:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827956.583151:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827956/real 1787827956] req@ffff8ab3d0a43c00 x1874671280701824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827956.583173:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280701824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827956.583191:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827956.583198:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827956.583204:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280701824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827956.583216:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280701952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827956.583251:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827956.583274:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827956/real 1787827956] req@ffff8ab3d0a40000 x1874671280701952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827956.583308:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280701952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827956.583327:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827956.583334:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827956.583341:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280701952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827957.606092:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827958.630107:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827959.654064:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827960.678141:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827961.701965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827961.701976:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827961.701980:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827961.701990:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2511 00000100:00080000:3.0:1787827961.702000:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827961.702009:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827961.702018:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827961.702046:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280702080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827961.702067:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827961.702070:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827961.702075:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827961.702078:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2511 00000100:00080000:3.0:1787827961.702080:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827961.702084:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827961.702092:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827961.702104:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280702208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827961.702116:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280702080:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827961.702118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827961.702125:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827961.702127:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827961.702133:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2511 00000100:00080000:3.0:1787827961.702137:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827961.702141:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827961.702146:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827961.702158:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280702336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827961.702165:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827961.702168:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787827961.702172:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827961.702172:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827961.702179:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2511 00000100:00080000:3.0:1787827961.702183:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827961.702186:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827961.702191:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827961.702200:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280702464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827961.702206:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827961.702208:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827961.702210:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827961.702213:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2511 00000100:00080000:3.0:1787827961.702215:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827961.702217:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827961.702221:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827961.702223:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827961/real 1787827961] req@ffff8ab3d0a41e00 x1874671280702080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827961.702232:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280702592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827961.702244:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280702080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827961.702253:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827961.702257:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827961.702260:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280702080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827961.702277:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280702208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827961.702293:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827961.702301:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827961/real 1787827961] req@ffff8ab3d0a43480 x1874671280702208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827961.702309:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280702208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827961.702319:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827961.702321:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827961.702323:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280702208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827961.702327:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280702336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827961.702336:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827961.702349:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827961/real 1787827961] req@ffff8ab3d0a43840 x1874671280702336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827961.702357:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280702336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827961.702362:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827961.702364:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827961.702366:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280702336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827961.702370:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280702464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827961.702381:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827961.702387:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827961/real 1787827961] req@ffff8ab3d0a430c0 x1874671280702464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827961.702394:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280702464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827961.702400:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827961.702401:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827961.702404:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280702464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827961.702408:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280702592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827961.702417:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827961.702422:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827961/real 1787827961] req@ffff8ab3d0a40000 x1874671280702592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827961.702429:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280702592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827961.702435:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827961.702437:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827961.702438:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280702592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827962.726116:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827963.750088:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827964.774083:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827965.798086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827966.822080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1787827966.822083:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827966.822095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827966.822124:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827966.822141:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2516 00000100:00080000:0.0:1787827966.822156:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827966.822169:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787827966.822227:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280702720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827966.822272:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827966.822277:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827966.822280:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827966.822285:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2516 00000100:00080000:0.0:1787827966.822290:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827966.822297:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827966.822310:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827966.822329:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280702848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827966.822344:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827966.822352:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787827966.822355:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280702720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787827966.822356:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827966.822370:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2516 00000100:00080000:0.0:1787827966.822376:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827966.822381:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827966.822387:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827966.822405:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280702976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827966.822417:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827966.822422:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827966.822425:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827966.822429:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2516 00000100:00080000:0.0:1787827966.822445:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827966.822451:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:3.0:1787827966.822460:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787827966.822463:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787827966.822483:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280703104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827966.822495:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827966.822501:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827966.822503:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827966.822509:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2516 00000100:00080000:0.0:1787827966.822515:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827966.822519:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787827966.822525:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827966.822542:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280703232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827966.822549:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827966/real 1787827966] req@ffff8ab3f075cf00 x1874671280702720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827966.822594:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280702720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827966.822618:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827966.822625:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827966.822630:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280702720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827966.822660:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280702848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827966.822688:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827966.822704:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827966/real 1787827966] req@ffff8ab3f075e580 x1874671280702848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827966.822719:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280702848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827966.822730:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827966.822734:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827966.822738:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280702848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827966.822745:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280702976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787827966.822763:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827966.822773:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827966/real 1787827966] req@ffff8ab3f075ed00 x1874671280702976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827966.822787:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280702976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827966.822797:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827966.822826:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827966.822832:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280702976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827966.822841:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280703104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827966.822869:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827966.822912:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827966/real 1787827966] req@ffff8ab3f075f480 x1874671280703104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827966.822930:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280703104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827966.822942:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827966.822946:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827966.822950:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280703104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787827966.822958:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280703232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787827966.822982:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827966.822995:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827966/real 1787827966] req@ffff8ab3f075da40 x1874671280703232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827966.823008:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280703232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827966.823020:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827966.823023:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787827966.823027:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280703232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827967.846072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827968.870076:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827969.894141:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827970.918106:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827971.942068:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827971.942084:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827971.942092:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827971.942109:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2521 00000100:00080000:3.0:1787827971.942126:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827971.942139:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827971.942172:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827971.942208:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280703360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827971.942252:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827971.942257:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827971.942260:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827971.942264:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2521 00000100:00080000:3.0:1787827971.942268:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827971.942274:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827971.942300:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827971.942318:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280703360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827971.942323:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280703488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827971.942342:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827971.942347:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827971.942349:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827971.942356:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2521 00000100:00080000:3.0:1787827971.942361:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827971.942365:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827971.942371:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827971.942387:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280703616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827971.942408:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827971.942412:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827971.942416:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827971.942420:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2521 00000100:00080000:3.0:1787827971.942424:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827971.942428:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827971.942435:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787827971.942445:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787827971.942455:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280703744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827971.942468:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827971.942474:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827971.942477:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827971.942481:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2521 00000100:00080000:3.0:1787827971.942490:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827971.942494:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827971.942499:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827971.942516:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280703872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827971.942557:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827971/real 1787827971] req@ffff8ab3d0a43c00 x1874671280703360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827971.942589:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280703360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827971.942606:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827971.942611:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827971.942617:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280703360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827971.942653:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280703488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827971.942686:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827971.942702:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827971/real 1787827971] req@ffff8ab3d0a403c0 x1874671280703488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827971.942716:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280703488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827971.942727:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827971.942731:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827971.942735:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280703488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827971.942742:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280703616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827971.942761:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827971.942806:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827971/real 1787827971] req@ffff8ab3d0a421c0 x1874671280703616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827971.942821:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280703616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827971.942831:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827971.942835:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827971.942839:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280703616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827971.942846:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280703744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827971.942871:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827971.942882:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827971/real 1787827971] req@ffff8ab3d0a412c0 x1874671280703744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827971.942936:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280703744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827971.942948:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827971.942952:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827971.942957:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280703744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827971.942964:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280703872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827971.942987:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827971.942999:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827971/real 1787827971] req@ffff8ab3d0a40000 x1874671280703872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827971.943012:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280703872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827971.943024:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827971.943027:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827971.943030:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280703872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827972.966063:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827973.990144:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827975.014007:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827976.038086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827977.062040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827977.062072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827977.062079:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827977.062094:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2526 00000100:00080000:3.0:1787827977.062112:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827977.062123:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827977.062183:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827977.062205:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280704000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827977.062254:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827977.062258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827977.062260:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827977.062264:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2526 00000100:00080000:3.0:1787827977.062267:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827977.062271:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827977.062301:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827977.062325:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280704128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827977.062335:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280704000:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827977.062373:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827977.062382:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827977.062386:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827977.062400:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2526 00000100:00080000:3.0:1787827977.062414:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787827977.062422:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827977.062427:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827977.062440:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827977.062462:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280704256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827977.062476:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827977.062481:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787827977.062487:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827977/real 1787827977] req@ffff8ab3d0a430c0 x1874671280704000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827977.062488:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827977.062506:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2526 00000100:00080000:3.0:1787827977.062523:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827977.062533:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280704000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827977.062534:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827977.062552:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:0.0:1787827977.062555:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827977.062563:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827977.062567:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280704000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827977.062575:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280704384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827977.062591:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827977.062600:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787827977.062601:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280704128:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787827977.062607:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827977.062633:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2526 00000100:00080000:3.0:1787827977.062640:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787827977.062647:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827977.062652:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827977.062660:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827977.062667:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827977/real 1787827977] req@ffff8ab3d0a43840 x1874671280704128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827977.062678:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280704512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827977.062681:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280704128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827977.062748:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827977.062751:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827977.062756:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280704128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827977.062762:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280704256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827977.062782:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827977.062790:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827977/real 1787827977] req@ffff8ab3d0a43480 x1874671280704256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827977.062798:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280704256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827977.062804:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827977.062806:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827977.062808:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280704256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827977.062823:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280704384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827977.062834:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827977.062840:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827977/real 1787827977] req@ffff8ab3d0a41e00 x1874671280704384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827977.062847:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280704384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827977.062855:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827977.062857:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827977.062859:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280704384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827977.062870:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280704512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827977.062880:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827977.062932:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827977/real 1787827977] req@ffff8ab3d0a430c0 x1874671280704512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827977.062940:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280704512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827977.062946:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827977.062948:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827977.062950:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280704512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827978.086161:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827979.110207:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827980.134131:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827981.158131:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827982.182073:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827982.182097:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827982.182106:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827982.182136:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2531 00000100:00080000:3.0:1787827982.182155:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787827982.182156:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827982.182173:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827982.182261:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280704640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827982.182320:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827982.182328:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827982.182332:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827982.182340:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2531 00000100:00080000:3.0:1787827982.182348:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827982.182357:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827982.182386:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827982.182419:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280704640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827982.182421:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280704768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827982.182459:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827982.182466:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827982.182470:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827982.182490:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2531 00000100:00080000:3.0:1787827982.182499:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827982.182510:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827982.182521:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787827982.182552:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827982.182657:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827982/real 1787827982] req@ffff8ab3d0a40000 x1874671280704640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827982.182727:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280704640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827982.182759:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827982.182769:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827982.182779:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280704640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827982.182809:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280704896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827982.182839:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280704768:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787827982.182853:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827982.182863:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827982.182869:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827982.182880:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2531 00000100:00080000:3.0:1787827982.182949:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827982.182960:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787827982.182971:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787827982.182974:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827982.183011:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280705024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827982.183033:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827982/real 1787827982] req@ffff8ab3d0a412c0 x1874671280704768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827982.183034:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827982.183053:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827982.183064:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827982.183078:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2531 00000100:00100000:0.0:1787827982.183084:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280704768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827982.183091:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827982.183106:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:0.0:1787827982.183119:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787827982.183125:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:0.0:1787827982.183129:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827982.183143:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280704768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787827982.183167:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280705152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827982.183175:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280704896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827982.183227:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827982.183255:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827982/real 1787827982] req@ffff8ab3d0a421c0 x1874671280704896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827982.183280:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280704896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827982.183298:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827982.183306:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827982.183316:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280704896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827982.183331:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280705024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827982.183370:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827982.183396:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827982/real 1787827982] req@ffff8ab3d0a403c0 x1874671280705024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827982.183422:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280705024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827982.183444:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827982.183451:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827982.183459:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280705024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827982.183495:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280705152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827982.183541:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827982.183566:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827982/real 1787827982] req@ffff8ab3d0a43c00 x1874671280705152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827982.183593:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280705152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827982.183610:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827982.183614:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827982.183619:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280705152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827983.206117:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827984.230149:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827985.254117:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827986.278061:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827987.302009:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827987.302022:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827987.302028:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827987.302043:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2536 00000100:00080000:3.0:1787827987.302058:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827987.302071:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787827987.302106:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827987.302184:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280705280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827987.302222:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827987.302231:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827987.302233:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827987.302238:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2536 00000100:00080000:3.0:1787827987.302244:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827987.302250:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827987.302262:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827987.302281:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280705408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827987.302294:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827987.302299:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827987.302302:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827987.302317:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2536 00000100:00080000:3.0:1787827987.302321:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827987.302326:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827987.302331:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827987.302344:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280705536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827987.302356:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280705280:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827987.302364:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827987.302371:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827987.302373:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827987.302380:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2536 00000100:00080000:3.0:1787827987.302384:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827987.302389:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827987.302397:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827987.302415:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280705664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827987.302428:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827987.302433:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827987.302435:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827987.302439:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2536 00000100:00080000:3.0:1787827987.302444:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787827987.302448:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827987.302465:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827987.302473:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827987.302491:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280705792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827987.302535:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827987/real 1787827987] req@ffff8ab3d0a43c00 x1874671280705280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827987.302567:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280705280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827987.302582:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827987.302587:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827987.302593:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280705280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827987.302602:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280705408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827987.302629:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827987.302643:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827987/real 1787827987] req@ffff8ab3d0a403c0 x1874671280705408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827987.302687:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280705408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827987.302709:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827987.302713:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827987.302717:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280705408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827987.302749:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280705536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827987.302773:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827987.302785:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827987/real 1787827987] req@ffff8ab3d0a421c0 x1874671280705536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827987.302800:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280705536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827987.302810:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827987.302814:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827987.302819:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280705536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827987.302826:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280705664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827987.302845:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827987.302856:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827987/real 1787827987] req@ffff8ab3d0a412c0 x1874671280705664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827987.302869:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280705664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827987.302881:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827987.302924:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827987.302929:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280705664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827987.302936:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280705792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827987.302968:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827987.302983:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827987/real 1787827987] req@ffff8ab3d0a40000 x1874671280705792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827987.302997:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280705792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827987.303008:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827987.303012:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827987.303016:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280705792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827988.326044:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827989.350138:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827990.374142:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827991.398083:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827992.422056:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827992.422070:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827992.422077:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827992.422092:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2542 00000100:00080000:3.0:1787827992.422107:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827992.422116:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827992.422175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280705920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827992.422187:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827992.422212:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827992.422219:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827992.422222:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827992.422226:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2542 00000100:00080000:3.0:1787827992.422246:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827992.422250:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827992.422263:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827992.422279:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280706048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827992.422291:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827992.422311:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827992.422315:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827992.422324:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2542 00000100:00080000:3.0:1787827992.422332:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827992.422336:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280705920:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787827992.422348:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827992.422361:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827992.422388:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280706176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827992.422401:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827992.422405:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827992.422408:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827992.422412:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2542 00000100:00080000:3.0:1787827992.422428:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827992.422439:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787827992.422441:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787827992.422467:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827992.422490:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280706304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827992.422507:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827992.422528:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827992.422531:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827992.422539:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2542 00000100:00100000:0.0:1787827992.422541:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827992/real 1787827992] req@ffff8ab3d0a430c0 x1874671280705920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827992.422552:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827992.422573:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827992.422594:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787827992.422604:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280705920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827992.422617:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280706432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827992.422672:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827992.422699:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827992.422709:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280705920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827992.422733:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280706048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827992.422776:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827992.422795:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827992/real 1787827992] req@ffff8ab3d0a41e00 x1874671280706048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827992.422820:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280706048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827992.422831:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827992.422835:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827992.422839:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280706048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827992.422873:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280706176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827992.422943:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827992.422959:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827992/real 1787827992] req@ffff8ab3d0a43480 x1874671280706176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827992.422974:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280706176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827992.422993:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827992.422997:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827992.423001:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280706176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827992.423009:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280706304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827992.423035:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827992.423055:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827992/real 1787827992] req@ffff8ab3d0a43840 x1874671280706304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827992.423077:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280706304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827992.423096:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827992.423103:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827992.423110:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280706304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827992.423123:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280706432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827992.423158:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827992.423174:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827992/real 1787827992] req@ffff8ab3d0a40000 x1874671280706432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827992.423189:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280706432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827992.423201:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827992.423205:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827992.423210:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280706432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827993.446087:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827994.470065:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827995.494108:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827996.518136:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827997.542006:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827997.542018:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827997.542024:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827997.542049:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2547 00000100:00080000:3.0:1787827997.542062:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827997.542072:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787827997.542119:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280706560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827997.542125:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827997.542160:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827997.542166:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827997.542168:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827997.542172:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2547 00000100:00080000:3.0:1787827997.542177:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827997.542181:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827997.542194:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827997.542209:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280706688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827997.542222:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827997.542227:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827997.542229:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827997.542234:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2547 00000100:00080000:3.0:1787827997.542237:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827997.542241:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827997.542247:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827997.542254:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280706560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787827997.542265:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280706816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827997.542282:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827997.542286:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827997.542288:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827997.542294:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2547 00000100:00080000:3.0:1787827997.542298:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827997.542302:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827997.542309:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787827997.542325:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280706944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827997.542337:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827997.542341:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827997.542344:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827997.542349:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2547 00000800:00000100:0.0:1787827997.542358:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787827997.542358:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827997.542370:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787827997.542377:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787827997.542392:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280707072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827997.542465:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827997/real 1787827997] req@ffff8ab3d0a412c0 x1874671280706560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827997.542492:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280706560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827997.542506:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827997.542511:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827997.542518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280706560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787827997.542530:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280706688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827997.542562:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827997.542578:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827997/real 1787827997] req@ffff8ab3d0a421c0 x1874671280706688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827997.542623:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280706688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827997.542651:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827997.542655:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827997.542660:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280706688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827997.542689:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280706816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787827997.542714:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827997.542727:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827997/real 1787827997] req@ffff8ab3d0a403c0 x1874671280706816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827997.542740:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280706816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827997.542750:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827997.542753:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827997.542775:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280706816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787827997.542783:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280706944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827997.542805:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827997.542816:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827997/real 1787827997] req@ffff8ab3d0a43c00 x1874671280706944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827997.542830:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280706944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827997.542841:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827997.542845:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827997.542849:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280706944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787827997.542855:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280707072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787827997.542877:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787827997.542897:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827997/real 1787827997] req@ffff8ab3d0a40000 x1874671280707072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827997.542984:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280707072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827997.543005:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827997.543012:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787827997.543019:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280707072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787827998.566070:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827999.590179:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828000.614077:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828001.638158:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828002.662034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828002.662048:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828002.662055:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828002.662070:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2552 00000100:00080000:0.0:1787828002.662084:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828002.662095:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828002.662102:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828002.662153:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280707200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828002.662224:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828002.662229:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828002.662232:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828002.662237:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2552 00000100:00080000:0.0:1787828002.662243:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828002.662247:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828002.662262:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828002.662282:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280707328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828002.662295:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828002.662300:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828002.662304:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828002.662310:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2552 00000100:00080000:0.0:1787828002.662314:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828002.662320:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828002.662325:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828002.662339:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280707456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828002.662351:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828002.662356:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828002.662359:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828002.662364:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2552 00000100:00080000:0.0:1787828002.662368:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828002.662379:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828002.662380:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280707200:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787828002.662385:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828002.662411:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280707584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828002.662434:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828002.662441:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828002.662445:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828002.662454:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2552 00000100:00080000:0.0:1787828002.662491:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828002.662501:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:3.0:1787828002.662511:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828002.662514:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828002.662539:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280707712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828002.662628:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828002/real 1787828002] req@ffff8ab3f075cb40 x1874671280707200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828002.662655:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280707200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828002.662670:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828002.662676:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787828002.662682:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280707200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828002.662693:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280707328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787828002.662723:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828002.662738:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828002/real 1787828002] req@ffff8ab3f075da40 x1874671280707328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828002.662753:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280707328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828002.662773:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828002.662777:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787828002.662782:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280707328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787828002.662788:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280707456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787828002.662807:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828002.662817:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828002/real 1787828002] req@ffff8ab3f075f480 x1874671280707456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828002.662831:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280707456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828002.662841:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828002.662844:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787828002.662847:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280707456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787828002.662927:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280707584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787828002.662969:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828002.662981:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828002/real 1787828002] req@ffff8ab3f075ed00 x1874671280707584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828002.662992:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280707584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828002.663002:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828002.663006:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787828002.663010:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280707584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787828002.663016:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280707712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787828002.663031:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828002.663039:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828002/real 1787828002] req@ffff8ab3f075e580 x1874671280707712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828002.663050:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280707712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828002.663060:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828002.663062:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787828002.663065:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280707712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828003.686057:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828004.710093:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828005.734091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828006.758142:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828007.782015:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828007.782030:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828007.782037:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828007.782052:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2557 00000100:00080000:1.0:1787828007.782069:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828007.782083:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828007.782171:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280707840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828007.782212:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828007.782217:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828007.782219:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828007.782224:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2557 00000100:00080000:1.0:1787828007.782229:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828007.782234:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828007.782249:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828007.782285:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280707968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828007.782296:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828007.782301:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828007.782304:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828007.782309:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2557 00000100:00080000:1.0:1787828007.782312:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828007.782317:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828007.782324:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828007.782342:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280708096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828007.782351:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280707840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787828007.782379:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828007.782384:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828007.782396:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828007.782406:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2557 00000100:00080000:1.0:1787828007.782410:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828007.782418:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828007.782424:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828007.782440:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280708224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828007.782453:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828007.782458:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828007.782461:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828007.782467:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2557 00000100:00080000:1.0:1787828007.782474:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828007.782481:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828007.782488:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:3.0:1787828007.782492:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828007.782551:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280708352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828007.782624:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828007/real 1787828007] req@ffff8ab3d22ff480 x1874671280707840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828007.782654:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280707840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1787828007.782664:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828007.782676:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828007.782690:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787828007.782699:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280707840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828007.782728:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280707968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787828007.782789:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828007.782818:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828007/real 1787828007] req@ffff8ab3d22fda40 x1874671280707968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828007.782845:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280707968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828007.782863:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828007.782870:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787828007.782875:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280707968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787828007.782959:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280708096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787828007.782991:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828007.783006:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828007/real 1787828007] req@ffff8ab3d22fcb40 x1874671280708096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828007.783022:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280708096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828007.783033:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828007.783039:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787828007.783046:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280708096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787828007.783059:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280708224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787828007.783096:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828007.783119:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828007/real 1787828007] req@ffff8ab3d22fd680 x1874671280708224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828007.783144:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280708224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828007.783162:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828007.783167:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787828007.783171:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280708224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787828007.783180:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280708352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787828007.783207:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828007.783220:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828007/real 1787828007] req@ffff8ab3d22fc000 x1874671280708352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828007.783234:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280708352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828007.783249:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828007.783256:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787828007.783262:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280708352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828008.806155:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828009.830135:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828010.854022:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828011.878128:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828012.902101:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828012.902114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828012.902120:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828012.902134:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2562 00010000:00010000:2.0:1787828012.902142:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828012.902149:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828012.902163:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828012.902214:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280708480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828012.902267:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828012.902273:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828012.902275:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828012.902280:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2562 00000100:00080000:3.0:1787828012.902284:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828012.902289:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828012.902314:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828012.902330:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280708608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828012.902349:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828012.902354:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828012.902356:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828012.902360:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2562 00000100:00080000:3.0:1787828012.902364:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828012.902368:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828012.902373:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828012.902385:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280708736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828012.902399:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828012.902403:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828012.902406:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828012.902411:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2562 00000100:00080000:3.0:1787828012.902415:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828012.902420:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828012.902430:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828012.902438:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280708480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828012.902451:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280708864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828012.902467:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828012.902471:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828012.902474:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828012.902506:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2562 00000100:00080000:3.0:1787828012.902514:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828012.902520:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828012.902529:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828012.902547:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280708992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828012.902563:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828012.902627:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828012/real 1787828012] req@ffff8ab3d0a43840 x1874671280708480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828012.902652:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280708480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828012.902666:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828012.902671:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828012.902677:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280708480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828012.902687:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280708608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828012.902713:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828012.902728:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828012/real 1787828012] req@ffff8ab3d0a43480 x1874671280708608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828012.902743:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280708608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828012.902753:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828012.902757:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828012.902761:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280708608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828012.902768:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280708736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828012.902805:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828012.902819:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828012/real 1787828012] req@ffff8ab3d0a41e00 x1874671280708736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828012.902833:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280708736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828012.902844:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828012.902847:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828012.902852:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280708736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828012.902949:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280708864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828012.902992:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828012.903014:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828012/real 1787828012] req@ffff8ab3d0a430c0 x1874671280708864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828012.903031:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280708864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828012.903043:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828012.903047:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828012.903051:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280708864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828012.903058:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280708992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828012.903082:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828012.903098:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828012/real 1787828012] req@ffff8ab3d0a40000 x1874671280708992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828012.903121:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280708992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828012.903132:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828012.903135:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828012.903139:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280708992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828013.926183:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828014.950169:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828015.974141:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828016.998033:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828018.022034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828018.022047:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828018.022067:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828018.022082:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2567 00000100:00080000:1.0:1787828018.022094:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828018.022107:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828018.022166:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280709120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828018.022219:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828018.022226:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828018.022230:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828018.022238:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2567 00000100:00080000:1.0:1787828018.022245:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828018.022253:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828018.022269:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828018.022300:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280709248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828018.022329:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280709120:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787828018.022336:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828018.022341:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828018.022343:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828018.022347:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2567 00000100:00080000:1.0:1787828018.022352:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828018.022356:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828018.022361:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828018.022394:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280709376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828018.022415:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828018.022420:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828018.022422:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828018.022428:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2567 00000100:00080000:1.0:1787828018.022434:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828018.022439:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828018.022485:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828018.022498:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280709504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828018.022508:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828018.022514:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828018.022516:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828018.022520:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2567 00000100:00080000:1.0:1787828018.022523:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828018.022528:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828018.022533:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280709248:192.168.203.124@tcp:38:kworker.0 00000008:00100000:1.0:1787828018.022534:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828018.022551:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280709632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828018.022597:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280709376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828018.022621:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280709504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828018.022646:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280709632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828018.022646:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828019.046019:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828020.070050:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828021.094208:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828021.094333:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787828021.094378:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828021.094394:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828021.094479:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828021.094494:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828021.094507:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828021.094520:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787828021.094550:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828018/real 1787828021] req@ffff8ab3d22fd680 x1874671280709248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828021.094591:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280709248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828021.094617:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828021.094624:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828021.094631:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280709248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828021.094642:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828018/real 1787828021] req@ffff8ab3d22fcb40 x1874671280709376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828021.094660:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280709376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828021.094672:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828021.094677:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828021.094681:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280709376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828021.094689:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828018/real 1787828021] req@ffff8ab3d22fda40 x1874671280709504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828021.094705:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280709504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828021.094717:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828021.094722:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828021.094727:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280709504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828021.094737:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828018/real 1787828021] req@ffff8ab3d22fc000 x1874671280709120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828021.094752:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280709120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828021.094763:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828021.094769:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828021.094773:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280709120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828021.094808:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828018/real 1787828021] req@ffff8ab3d22ff480 x1874671280709632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828021.094829:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280709632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828021.094843:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828021.094848:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828021.094853:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280709632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828022.118139:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828023.142011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828023.142024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828023.142029:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828023.142042:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2572 00000100:00080000:1.0:1787828023.142054:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828023.142064:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828023.142113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280709760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828023.142159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828023.142166:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828023.142170:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828023.142178:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2572 00000100:00080000:1.0:1787828023.142188:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828023.142199:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828023.142217:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828023.142238:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280709760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828023.142249:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280709888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828023.142270:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828023.142276:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828023.142279:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828023.142283:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2572 00000100:00080000:1.0:1787828023.142288:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828023.142294:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828023.142302:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828023.142321:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280710016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828023.142334:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828023.142338:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828023.142341:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828023.142346:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2572 00000100:00080000:1.0:1787828023.142350:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828023.142354:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828023.142362:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787828023.142366:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828023.142386:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280710144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828023.142451:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828023.142458:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828023.142462:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828023.142474:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2572 00000100:00100000:0.0:1787828023.142481:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828023/real 1787828023] req@ffff8ab3d22ff480 x1874671280709760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828023.142485:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828023.142501:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828023.142519:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828023.142545:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280709760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828023.142551:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280710272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828023.142608:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828023.142623:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828023.142635:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280709760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828023.142687:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280709888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828023.142746:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828023.142777:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828023/real 1787828023] req@ffff8ab3d22fc000 x1874671280709888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828023.142803:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280709888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828023.142821:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828023.142826:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828023.142833:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280709888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828023.142844:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280710016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828023.142886:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00010000:00010000:2.0:1787828023.142945:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828023.143028:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828023/real 1787828023] req@ffff8ab3d22fda40 x1874671280710016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828023.143055:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280710016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828023.143087:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828023.143095:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828023.143101:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280710016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828023.143115:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280710144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828023.143161:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828023.143186:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828023/real 1787828023] req@ffff8ab3d22fcb40 x1874671280710144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828023.143210:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280710144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828023.143230:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828023.143238:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828023.143245:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280710144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828023.143258:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280710272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828023.143300:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828023.143326:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828023/real 1787828023] req@ffff8ab3d22fd680 x1874671280710272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828023.143353:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280710272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828023.143374:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828023.143381:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828023.143389:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280710272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828024.166165:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828025.190072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828026.214172:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828027.238118:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828028.262040:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828028.262055:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828028.262062:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828028.262080:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2577 00000100:00080000:3.0:1787828028.262097:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828028.262111:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828028.262123:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828028.262172:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280710400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828028.262238:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828028.262247:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828028.262251:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828028.262259:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2577 00000100:00080000:3.0:1787828028.262268:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828028.262276:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828028.262293:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828028.262318:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280710528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828028.262334:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828028.262340:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828028.262342:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828028.262351:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2577 00000100:00100000:0.0:1787828028.262353:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280710400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828028.262356:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828028.262369:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828028.262402:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828028.262425:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280710656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828028.262447:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828028.262454:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828028.262458:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828028.262465:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2577 00000100:00080000:3.0:1787828028.262476:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828028.262489:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787828028.262491:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787828028.262506:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828028.262534:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280710784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828028.262548:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828028.262553:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828028.262556:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828028.262561:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2577 00000100:00080000:3.0:1787828028.262567:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828028.262573:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828028.262580:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828028.262587:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828028/real 1787828028] req@ffff8ab3d0a43c00 x1874671280710400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828028.262601:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280710912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828028.262629:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280710400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828028.262649:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828028.262655:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828028.262660:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280710400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828028.262672:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280710528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828028.262708:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828028.262730:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828028/real 1787828028] req@ffff8ab3d0a403c0 x1874671280710528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828028.262754:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280710528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828028.262770:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828028.262777:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828028.262784:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280710528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828028.262824:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280710656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828028.262860:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828028.262877:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828028/real 1787828028] req@ffff8ab3d0a421c0 x1874671280710656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828028.262938:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280710656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828028.262950:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828028.262954:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828028.262958:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280710656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828028.262966:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280710784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828028.262991:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828028.263005:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828028/real 1787828028] req@ffff8ab3d0a412c0 x1874671280710784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828028.263019:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280710784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828028.263030:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828028.263034:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828028.263037:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280710784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828028.263044:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280710912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828028.263061:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828028.263072:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828028/real 1787828028] req@ffff8ab3d0a40000 x1874671280710912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828028.263085:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280710912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828028.263097:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828028.263101:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828028.263104:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280710912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828029.286114:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828030.310124:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828031.334181:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828032.358196:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828033.382037:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828033.382055:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828033.382063:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828033.382079:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2583 00000100:00080000:3.0:1787828033.382092:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828033.382102:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828033.382127:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828033.382154:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280711040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828033.382193:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828033.382202:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828033.382205:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828033.382209:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2583 00000100:00080000:3.0:1787828033.382214:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828033.382220:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828033.382233:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828033.382252:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280711040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828033.382254:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280711168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828033.382304:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828033.382309:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828033.382313:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828033.382319:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2583 00000100:00080000:3.0:1787828033.382378:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828033.382385:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787828033.382395:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787828033.382396:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828033.382423:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280711296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828033.382438:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828033.382443:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828033.382446:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828033.382451:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2583 00000100:00080000:3.0:1787828033.382457:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828033.382463:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828033.382472:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828033.382477:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828033/real 1787828033] req@ffff8ab3d0a430c0 x1874671280711040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828033.382493:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280711424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828033.382508:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828033.382511:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280711040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828033.382514:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828033.382521:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828033.382536:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2583 00000100:00080000:3.0:1787828033.382544:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828033.382548:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828033.382552:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828033.382560:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828033.382574:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280711552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828033.382594:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828033.382603:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280711040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828033.382643:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280711168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828033.382677:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828033.382698:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828033/real 1787828033] req@ffff8ab3d0a41e00 x1874671280711168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828033.382721:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280711168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828033.382754:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828033.382773:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828033.382780:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280711168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828033.382795:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280711296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828033.382837:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828033.382862:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828033/real 1787828033] req@ffff8ab3d0a43480 x1874671280711296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828033.382880:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280711296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828033.382935:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828033.382940:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828033.382944:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280711296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828033.382953:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280711424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828033.382984:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828033.383000:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828033/real 1787828033] req@ffff8ab3d0a43840 x1874671280711424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828033.383015:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280711424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828033.383026:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828033.383030:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828033.383034:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280711424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828033.383041:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280711552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828033.383059:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828033.383070:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828033/real 1787828033] req@ffff8ab3d0a40000 x1874671280711552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828033.383085:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280711552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828033.383096:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828033.383100:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828033.383103:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280711552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828034.406066:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828035.430116:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828036.454069:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828037.478127:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828038.502006:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828038.502017:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828038.502023:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828038.502035:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2588 00000100:00080000:3.0:1787828038.502045:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828038.502061:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828038.502109:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280711680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828038.502112:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828038.502154:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828038.502161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828038.502164:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828038.502170:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2588 00000100:00080000:3.0:1787828038.502177:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828038.502184:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828038.502197:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828038.502220:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280711808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828038.502238:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828038.502242:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280711680:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828038.502245:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828038.502252:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828038.502263:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2588 00000100:00080000:3.0:1787828038.502269:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828038.502276:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828038.502284:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828038.502327:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280711936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828038.502345:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828038.502351:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828038.502353:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828038.502360:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2588 00000800:00000100:0.0:1787828038.502368:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828038.502369:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828038.502381:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828038.502390:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828038.502407:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280712064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828038.502424:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828038.502428:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828038.502430:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828038.502434:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2588 00000100:00080000:3.0:1787828038.502438:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828038.502443:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828038.502447:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828038.502452:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828038/real 1787828038] req@ffff8ab3d0a412c0 x1874671280711680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828038.502463:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280712192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828038.502515:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280711680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828038.502536:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828038.502543:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828038.502550:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280711680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828038.502587:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280711808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828038.502626:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828038.502650:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828038/real 1787828038] req@ffff8ab3d0a421c0 x1874671280711808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828038.502671:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280711808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828038.502713:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828038.502720:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828038.502727:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280711808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828038.502738:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280711936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828038.502772:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828038.502791:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828038/real 1787828038] req@ffff8ab3d0a403c0 x1874671280711936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828038.502811:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280711936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828038.502824:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828038.502830:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828038.502836:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280711936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828038.502845:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280712064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828038.502877:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828038.502948:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828038/real 1787828038] req@ffff8ab3d0a43c00 x1874671280712064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828038.502970:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280712064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828038.503000:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828038.503006:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828038.503012:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280712064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828038.503022:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280712192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828038.503055:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828038.503074:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828038/real 1787828038] req@ffff8ab3d0a40000 x1874671280712192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828038.503094:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280712192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828038.503109:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828038.503114:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828038.503120:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280712192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828039.526062:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828040.550107:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828041.574119:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828042.598131:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828043.622071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1787828043.622087:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828043.622088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828043.622101:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828043.622133:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2593 00000100:00080000:3.0:1787828043.622149:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828043.622160:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828043.622210:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280712320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828043.622246:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828043.622251:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828043.622254:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828043.622292:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2593 00000100:00080000:3.0:1787828043.622300:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828043.622307:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828043.622322:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828043.622335:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280712320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828043.622347:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280712448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828043.622365:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828043.622369:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828043.622373:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828043.622378:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2593 00000100:00080000:3.0:1787828043.622382:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828043.622388:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828043.622395:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828043.622409:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280712576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828043.622422:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828043.622432:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828043.622433:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828043.622441:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828043.622448:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2593 00000100:00080000:3.0:1787828043.622455:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828043.622461:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828043.622467:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828043.622484:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280712704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828043.622497:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828043.622505:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828043/real 1787828043] req@ffff8ab3d0a43840 x1874671280712320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828043.622506:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828043.622512:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828043.622527:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2593 00000100:00080000:3.0:1787828043.622537:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828043.622543:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828043.622551:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280712320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828043.622552:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828043.622578:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280712832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828043.622595:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828043.622605:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828043.622613:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280712320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828043.622644:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280712448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828043.622673:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828043.622691:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828043/real 1787828043] req@ffff8ab3d0a43480 x1874671280712448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828043.622706:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280712448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828043.622717:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828043.622721:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828043.622725:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280712448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828043.622732:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280712576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828043.622749:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828043.622760:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828043/real 1787828043] req@ffff8ab3d0a41e00 x1874671280712576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828043.622773:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280712576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828043.622783:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828043.622786:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828043.622790:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280712576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828043.622797:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280712704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828043.622814:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828043.622825:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828043/real 1787828043] req@ffff8ab3d0a430c0 x1874671280712704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828043.622839:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280712704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828043.622850:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828043.622854:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828043.622858:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280712704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828043.622864:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280712832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828043.622881:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828043.622931:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828043/real 1787828043] req@ffff8ab3d0a40000 x1874671280712832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828043.622946:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280712832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828043.622958:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828043.622962:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828043.622966:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280712832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828044.646101:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828045.670070:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828046.694123:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828047.718118:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828048.742018:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828048.742053:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828048.742061:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828048.742075:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2598 00000100:00080000:3.0:1787828048.742088:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828048.742098:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828048.742110:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828048.742149:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280712960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828048.742188:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828048.742193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828048.742196:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828048.742200:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2598 00000100:00080000:3.0:1787828048.742205:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828048.742209:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828048.742221:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828048.742269:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280713088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828048.742297:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828048.742302:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280712960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828048.742304:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828048.742312:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828048.742322:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2598 00000100:00080000:3.0:1787828048.742328:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828048.742334:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828048.742342:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828048.742363:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280713216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828048.742380:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828048.742388:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828048.742392:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828048.742400:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828048.742407:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2598 00000100:00080000:3.0:1787828048.742420:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828048.742431:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828048.742443:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828048.742469:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828048/real 1787828048] req@ffff8ab3d0a43c00 x1874671280712960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828048.742477:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280713344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828048.742514:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828048.742521:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280712960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828048.742523:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828048.742530:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828048.742551:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2598 00000100:00080000:0.0:1787828048.742554:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828048.742567:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787828048.742568:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828048.742575:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280712960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828048.742588:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828048.742602:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828048.742612:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280713088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787828048.742635:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280713472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828048.742651:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828048.742670:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828048/real 1787828048] req@ffff8ab3d0a403c0 x1874671280713088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828048.742699:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280713088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828048.742711:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828048.742717:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828048.742722:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280713088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828048.742730:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280713216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828048.742748:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828048.742759:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828048/real 1787828048] req@ffff8ab3d0a421c0 x1874671280713216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828048.742773:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280713216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828048.742783:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828048.742786:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828048.742790:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280713216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828048.742797:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280713344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828048.742815:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828048.742826:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828048/real 1787828048] req@ffff8ab3d0a412c0 x1874671280713344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828048.742840:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280713344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828048.742851:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828048.742854:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828048.742858:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280713344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828048.742877:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280713472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828048.742941:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828048.742956:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828048/real 1787828048] req@ffff8ab3d0a43c00 x1874671280713472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828048.742971:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280713472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828048.742982:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828048.742987:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828048.742991:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280713472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828049.766144:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828050.790003:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828051.814204:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828052.838106:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828053.862040:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828053.862057:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828053.862064:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828053.862082:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2603 00010000:00010000:2.0:1787828053.862095:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828053.862098:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828053.862117:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828053.862178:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280713600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828053.862254:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828053.862262:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828053.862266:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828053.862274:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2603 00000100:00080000:3.0:1787828053.862283:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828053.862293:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828053.862310:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828053.862323:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280713600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828053.862341:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280713728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828053.862360:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828053.862369:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828053.862371:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828053.862378:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2603 00000100:00080000:3.0:1787828053.862405:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828053.862406:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828053.862420:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828053.862431:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828053.862452:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280713856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828053.862466:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828053.862471:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828053.862474:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828053.862480:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828053/real 1787828053] req@ffff8ab3d0a40000 x1874671280713600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828053.862485:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2603 00000100:00080000:3.0:1787828053.862494:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828053.862505:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828053.862517:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828053.862547:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280713600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828053.862548:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280713984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828053.862589:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828053.862604:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828053.862607:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828053.862610:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828053.862613:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2603 00000100:00080000:3.0:1787828053.862621:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828053.862622:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787828053.862633:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828053.862636:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280713600:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787828053.862644:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828053.862662:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280714112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828053.862680:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280713728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828053.862734:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828053.862785:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828053/real 1787828053] req@ffff8ab3d0a430c0 x1874671280713728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828053.862814:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280713728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828053.862829:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828053.862834:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828053.862838:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280713728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828053.862847:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280713856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828053.862877:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828053.862932:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828053/real 1787828053] req@ffff8ab3d0a41e00 x1874671280713856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828053.862958:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280713856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828053.862975:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828053.862982:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828053.862991:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280713856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828053.863002:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280713984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828053.863067:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828053.863085:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828053/real 1787828053] req@ffff8ab3d0a43480 x1874671280713984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828053.863100:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280713984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828053.863113:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828053.863116:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828053.863129:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280713984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828053.863152:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280714112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828053.863174:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828053.863186:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828053/real 1787828053] req@ffff8ab3d0a43840 x1874671280714112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828053.863233:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280714112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828053.863256:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828053.863259:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828053.863263:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280714112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828054.886129:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828055.910107:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828056.934103:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828057.958123:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828058.982008:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828058.982025:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828058.982032:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828058.982050:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2608 00000100:00080000:3.0:1787828058.982068:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828058.982081:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828058.982115:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828058.982154:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280714240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828058.982227:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828058.982232:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828058.982235:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828058.982241:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2608 00000100:00080000:3.0:1787828058.982246:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828058.982252:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828058.982269:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828058.982289:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280714368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828058.982304:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828058.982309:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828058.982311:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828058.982321:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2608 00000100:00080000:3.0:1787828058.982330:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828058.982339:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828058.982349:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828058.982372:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280714496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828058.982397:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280714240:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828058.982407:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828058.982420:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828058.982425:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828058.982436:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2608 00000100:00080000:3.0:1787828058.982441:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828058.982449:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828058.982458:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828058.982481:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280714624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828058.982494:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828058.982500:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828058.982503:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828058.982509:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2608 00000100:00080000:3.0:1787828058.982514:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828058.982525:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787828058.982532:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787828058.982535:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828058.982563:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280714752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828058.982647:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828058/real 1787828058] req@ffff8ab3d0a43840 x1874671280714240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828058.982687:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280714240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828058.982710:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828058.982718:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828058.982726:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280714240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828058.982744:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280714368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828058.982792:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828058.982817:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828058/real 1787828058] req@ffff8ab3d0a43480 x1874671280714368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828058.982844:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280714368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828058.982860:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828058.982867:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828058.982874:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280714368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828058.982989:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280714496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828058.983032:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828058.983057:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828058/real 1787828058] req@ffff8ab3d0a41e00 x1874671280714496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828058.983081:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280714496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828058.983111:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828058.983119:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828058.983126:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280714496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828058.983139:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280714624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828058.983205:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828058.983228:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828058/real 1787828058] req@ffff8ab3d0a430c0 x1874671280714624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828058.983252:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280714624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828058.983272:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828058.983278:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828058.983284:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280714624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828058.983297:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280714752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828058.983336:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828058.983358:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828058/real 1787828058] req@ffff8ab3d0a40000 x1874671280714752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828058.983382:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280714752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828058.983400:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828058.983405:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828058.983412:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280714752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828060.006182:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828061.030230:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828062.054168:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828063.078108:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828064.102083:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828064.102103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828064.102111:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828064.102152:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2613 00000100:00080000:0.0:1787828064.102171:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828064.102185:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828064.102220:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828064.102249:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280714880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828064.102302:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828064.102307:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828064.102310:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828064.102315:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2613 00000100:00080000:0.0:1787828064.102321:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828064.102327:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828064.102340:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828064.102363:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280715008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828064.102390:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828064.102395:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828064.102398:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828064.102405:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2613 00000100:00080000:0.0:1787828064.102411:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828064.102424:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828064.102428:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280714880:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787828064.102434:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828064.102468:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280715136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828064.102490:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828064.102496:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828064.102498:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828064.102504:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2613 00000100:00080000:0.0:1787828064.102510:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828064.102515:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828064.102525:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828064.102551:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280715264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828064.102572:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828064.102577:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828064.102588:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828064.102594:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828064.102606:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2613 00000100:00080000:0.0:1787828064.102616:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828064.102629:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828064.102641:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828064.102660:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828064/real 1787828064] req@ffff8ab3f075cf00 x1874671280714880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828064.102690:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280715392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828064.102694:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280714880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828064.102812:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828064.102821:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828064.102830:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280714880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828064.102843:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280715008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828064.102873:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828064.102960:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828064/real 1787828064] req@ffff8ab3f075f0c0 x1874671280715008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828064.102991:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280715008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828064.103005:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828064.103010:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828064.103015:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280715008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828064.103057:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280715136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828064.103088:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828064.103103:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828064/real 1787828064] req@ffff8ab3f075de00 x1874671280715136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828064.103119:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280715136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828064.103159:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828064.103164:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828064.103169:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280715136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828064.103178:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280715264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828064.103208:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828064.103222:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828064/real 1787828064] req@ffff8ab3f075f840 x1874671280715264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828064.103237:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280715264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828064.103259:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828064.103264:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828064.103268:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280715264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828064.103275:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280715392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828064.103294:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828064.103305:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828064/real 1787828064] req@ffff8ab3f075e940 x1874671280715392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828064.103318:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280715392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828064.103330:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828064.103334:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828064.103338:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280715392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828065.126204:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828066.150131:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828067.174219:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828068.198159:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828069.222143:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828069.222159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828069.222176:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828069.222183:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828069.222200:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2618 00000100:00080000:3.0:1787828069.222217:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828069.222229:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828069.222325:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280715520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828069.222398:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828069.222406:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828069.222411:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828069.222419:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2618 00000100:00080000:3.0:1787828069.222429:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828069.222440:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828069.222460:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828069.222495:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280715520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828069.222510:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280715648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828069.222531:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828069.222553:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828069.222556:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828069.222563:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2618 00000100:00080000:3.0:1787828069.222568:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828069.222573:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828069.222582:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828069.222598:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280715776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828069.222630:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828069.222647:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828069.222647:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828069.222658:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828069.222667:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2618 00000100:00080000:3.0:1787828069.222672:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828069.222698:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828069.222705:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828069.222726:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280715904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828069.222743:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828069.222748:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828069.222751:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828069.222758:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2618 00000100:00080000:3.0:1787828069.222768:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828069.222770:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828069/real 1787828069] req@ffff8ab3d0a43c00 x1874671280715520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828069.222780:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828069.222788:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828069.222812:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280716032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828069.222858:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280715520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828069.222884:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828069.222954:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828069.222964:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280715520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828069.223032:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280715648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828069.223083:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828069.223143:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828069/real 1787828069] req@ffff8ab3d0a412c0 x1874671280715648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828069.223171:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280715648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828069.223202:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828069.223209:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828069.223217:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280715648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828069.223231:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280715776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828069.223269:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828069.223291:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828069/real 1787828069] req@ffff8ab3d0a421c0 x1874671280715776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828069.223315:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280715776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828069.223332:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828069.223340:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828069.223347:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280715776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828069.223368:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280715904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828069.223410:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828069.223433:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828069/real 1787828069] req@ffff8ab3d0a403c0 x1874671280715904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828069.223458:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280715904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828069.223490:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828069.223497:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828069.223504:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280715904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828069.223518:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280716032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828069.223567:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828069.223589:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828069/real 1787828069] req@ffff8ab3d0a40000 x1874671280716032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828069.223625:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280716032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828069.223646:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828069.223653:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828069.223660:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280716032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828070.246127:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828071.270010:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828072.294180:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828073.318033:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828074.341995:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828074.342004:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828074.342009:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828074.342019:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2623 00000100:00080000:0.0:1787828074.342027:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787828074.342031:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828074.342035:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828074.342107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280716160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828074.342133:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828074.342136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828074.342138:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828074.342141:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2623 00000100:00080000:0.0:1787828074.342144:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828074.342147:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828074.342156:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828074.342170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280716288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828074.342189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828074.342194:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828074.342200:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828074.342200:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280716160:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828074.342213:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2623 00000100:00080000:0.0:1787828074.342218:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828074.342224:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828074.342231:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828074.342258:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280716416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828074.342270:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828074.342273:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787828074.342277:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828074.342278:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828074.342287:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2623 00000100:00080000:0.0:1787828074.342292:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828074.342296:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828074.342302:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828074.342314:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280716544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828074.342326:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828074.342329:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828074/real 1787828074] req@ffff8ab3f075c000 x1874671280716160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828074.342334:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828074.342339:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828074.342349:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2623 00000100:00100000:1.0:1787828074.342355:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280716160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828074.342357:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828074.342370:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787828074.342372:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828074.342377:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787828074.342379:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828074.342386:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280716160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828074.342396:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280716672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828074.342412:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280716288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828074.342432:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828074.342442:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828074/real 1787828074] req@ffff8ab3f075e940 x1874671280716288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828074.342451:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280716288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828074.342458:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828074.342461:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828074.342463:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280716288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828074.342467:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280716416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828074.342478:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828074.342485:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828074/real 1787828074] req@ffff8ab3f075f840 x1874671280716416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828074.342493:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280716416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828074.342499:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828074.342501:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828074.342504:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280716416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828074.342519:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280716544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828074.342530:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828074.342536:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828074/real 1787828074] req@ffff8ab3f075de00 x1874671280716544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828074.342544:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280716544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828074.342552:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828074.342554:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828074.342556:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280716544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828074.342560:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280716672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828074.342570:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828074.342576:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828074/real 1787828074] req@ffff8ab3f075f0c0 x1874671280716672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828074.342585:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280716672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828074.342591:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828074.342593:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828074.342595:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280716672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828075.366130:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828076.390167:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828077.414141:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828078.438145:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828079.462139:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828079.462194:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828079.462210:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828079.462219:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828079.462235:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2629 00000100:00080000:0.0:1787828079.462251:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828079.462263:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828079.462320:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280716800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828079.462366:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828079.462373:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828079.462377:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828079.462385:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2629 00000100:00080000:0.0:1787828079.462393:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828079.462404:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828079.462420:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828079.462433:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280716800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828079.462452:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280716928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828079.462477:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828079.462484:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828079.462489:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828079.462497:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2629 00000100:00080000:0.0:1787828079.462506:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828079.462515:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828079.462522:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828079.462532:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828079.462564:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280717056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828079.462587:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828079.462589:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828079/real 1787828079] req@ffff8ab3f075cf00 x1874671280716800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828079.462609:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828079.462617:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828079.462632:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2629 00000100:00100000:1.0:1787828079.462639:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280716800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828079.462645:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828079.462664:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787828079.462665:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828079.462679:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000008:00100000:0.0:1787828079.462680:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828079.462687:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280716800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828079.462717:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280717184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828079.462729:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280716928:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787828079.462760:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828079.462769:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787828079.462779:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828079.462780:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828079.462795:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2629 00000100:00080000:0.0:1787828079.462805:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828079.462807:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828079/real 1787828079] req@ffff8ab3f075e580 x1874671280716928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828079.462820:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828079.462841:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828079.462843:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280716928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828079.462863:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828079.462870:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828079.462881:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280716928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828079.462883:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280717312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828079.462924:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280717056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828079.462958:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828079.462986:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828079/real 1787828079] req@ffff8ab3f075ed00 x1874671280717056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828079.463008:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280717056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828079.463055:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828079.463060:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828079.463065:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280717056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828079.463087:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280717184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828079.463116:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828079.463137:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828079/real 1787828079] req@ffff8ab3f075f480 x1874671280717184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828079.463160:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280717184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828079.463179:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828079.463186:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828079.463194:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280717184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828079.463206:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280717312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828079.463243:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828079.463261:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828079/real 1787828079] req@ffff8ab3f075da40 x1874671280717312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828079.463276:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280717312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828079.463288:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828079.463292:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828079.463297:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280717312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828080.486121:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828081.510175:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828082.534161:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828083.558038:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828084.582065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828084.582078:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828084.582086:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1787828084.582101:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828084.582104:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2634 00000100:00080000:0.0:1787828084.582124:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828084.582138:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828084.582199:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280717440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828084.582244:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828084.582249:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828084.582252:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828084.582257:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2634 00000100:00080000:0.0:1787828084.582263:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828084.582270:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828084.582283:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828084.582307:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280717568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828084.582319:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280717440:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828084.582327:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828084.582338:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828084.582341:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828084.582347:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2634 00000100:00080000:0.0:1787828084.582351:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828084.582357:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828084.582373:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828084.582390:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280717696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828084.582403:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828084.582407:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828084.582410:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828084.582415:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2634 00000100:00080000:0.0:1787828084.582421:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828084.582426:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828084.582433:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828084.582448:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280717824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828084.582459:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828084.582464:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828084.582467:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828084.582473:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2634 00000100:00080000:0.0:1787828084.582478:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828084.582483:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828084.582490:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828084.582500:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280717568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828084.582510:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280717952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828084.582558:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280717696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828084.582586:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280717824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828084.582615:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280717952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828085.606140:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828086.630185:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828087.654148:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828087.654344:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:3.0:1787828087.654396:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828087.654416:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787828087.654459:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787828087.654470:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787828087.654479:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787828087.654488:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787828087.654576:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828084/real 1787828087] req@ffff8ab3f075e580 x1874671280717952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828087.654612:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280717952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828087.654629:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828087.654636:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828087.654641:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280717952:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828087.654651:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828084/real 1787828087] req@ffff8ab3f075da40 x1874671280717568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828087.654666:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280717568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828087.654676:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828087.654680:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828087.654684:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280717568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828087.654694:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828084/real 1787828087] req@ffff8ab3f075f480 x1874671280717696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828087.654727:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280717696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828087.654743:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828087.654748:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828087.654755:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280717696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828087.654767:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828084/real 1787828087] req@ffff8ab3f075ed00 x1874671280717824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828087.654791:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280717824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828087.654811:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828087.654817:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828087.654823:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280717824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828087.654837:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828084/real 1787828087] req@ffff8ab3f075cb40 x1874671280717440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828087.654869:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280717440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828087.654882:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828087.654939:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828087.654947:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280717440:192.168.203.124@tcp:250:kworker.0 00010000:00010000:2.0:1787828088.678176:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828089.702050:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828089.702062:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828089.702067:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828089.702096:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2639 00000100:00080000:0.0:1787828089.702107:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828089.702118:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828089.702191:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280718080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828089.702221:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828089.702256:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828089.702268:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828089.702271:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828089.702277:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2639 00000100:00080000:0.0:1787828089.702283:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828089.702289:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828089.702303:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828089.702328:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280718080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828089.702332:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280718208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828089.702359:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828089.702364:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828089.702366:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828089.702373:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2639 00000100:00080000:0.0:1787828089.702380:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828089.702387:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828089.702397:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828089.702420:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280718336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828089.702437:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828089.702444:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828089.702447:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828089.702458:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828089.702464:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2639 00000100:00080000:0.0:1787828089.702475:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828089.702483:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828089.702491:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828089.702510:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280718464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828089.702522:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828089.702526:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828089.702528:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828089.702533:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2639 00000100:00080000:0.0:1787828089.702538:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828089.702547:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828089/real 1787828089] req@ffff8ab3f075cf00 x1874671280718080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828089.702548:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828089.702558:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828089.702578:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280718592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828089.702597:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280718080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828089.702620:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828089.702628:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828089.702636:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280718080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828089.702673:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280718208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828089.702715:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828089.702740:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828089/real 1787828089] req@ffff8ab3f075f0c0 x1874671280718208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828089.702763:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280718208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828089.702778:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828089.702785:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828089.702791:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280718208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828089.702802:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280718336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828089.702838:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828089.702858:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828089/real 1787828089] req@ffff8ab3f075de00 x1874671280718336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828089.702879:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280718336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828089.702934:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828089.702941:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828089.702970:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280718336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828089.702983:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280718464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828089.703021:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828089.703042:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828089/real 1787828089] req@ffff8ab3f075f840 x1874671280718464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828089.703062:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280718464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828089.703078:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828089.703083:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828089.703088:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280718464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828089.703099:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280718592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828089.703140:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828089.703164:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828089/real 1787828089] req@ffff8ab3f075e940 x1874671280718592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828089.703183:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280718592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828089.703211:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828089.703216:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828089.703221:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280718592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828090.726135:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828091.750128:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828092.774171:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828093.798067:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828094.822093:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828094.822112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1787828094.822120:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828094.822121:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828094.822139:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2644 00000100:00080000:3.0:1787828094.822153:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828094.822162:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828094.822222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280718720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828094.822256:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828094.822261:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828094.822264:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828094.822269:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2644 00000100:00080000:3.0:1787828094.822274:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828094.822280:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828094.822294:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828094.822320:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280718720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828094.822334:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280718848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828094.822362:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828094.822369:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828094.822375:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828094.822384:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2644 00000100:00080000:3.0:1787828094.822393:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828094.822403:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828094.822408:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787828094.822422:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828094.822452:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280718976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828094.822473:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828094.822480:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787828094.822485:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828094/real 1787828094] req@ffff8ab3d0a430c0 x1874671280718720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828094.822488:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828094.822510:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2644 00000100:00080000:3.0:1787828094.822522:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828094.822527:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280718720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828094.822539:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828094.822549:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787828094.822567:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828094.822575:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787828094.822581:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280719104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828094.822589:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280718720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828094.822613:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828094.822618:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828094.822622:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828094.822630:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2644 00000100:00080000:3.0:1787828094.822636:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828094.822637:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280718848:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787828094.822649:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828094.822659:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828094.822677:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280719232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828094.822684:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828094.822710:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828094/real 1787828094] req@ffff8ab3d0a41e00 x1874671280718848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828094.822732:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280718848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828094.822745:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828094.822750:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828094.822755:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280718848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828094.822765:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280718976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828094.822785:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828094.822797:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828094/real 1787828094] req@ffff8ab3d0a43480 x1874671280718976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828094.822810:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280718976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828094.822820:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828094.822824:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828094.822827:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280718976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828094.822834:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280719104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828094.822853:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828094.822863:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828094/real 1787828094] req@ffff8ab3d0a43840 x1874671280719104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828094.822877:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280719104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828094.822945:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828094.822950:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828094.822954:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280719104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828094.822979:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280719232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828094.823004:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828094.823018:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828094/real 1787828094] req@ffff8ab3d0a430c0 x1874671280719232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828094.823033:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280719232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828094.823045:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828094.823048:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828094.823052:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280719232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828095.846198:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828096.870182:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828097.894112:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828098.918082:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828099.941994:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828099.942007:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828099.942013:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828099.942027:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2649 00000100:00080000:3.0:1787828099.942039:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828099.942051:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828099.942098:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828099.942102:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280719360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828099.942142:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828099.942147:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828099.942150:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828099.942154:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2649 00000100:00080000:3.0:1787828099.942159:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828099.942164:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828099.942177:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828099.942195:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280719488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828099.942205:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280719360:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828099.942218:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828099.942224:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828099.942226:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828099.942232:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2649 00000100:00080000:3.0:1787828099.942235:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828099.942241:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828099.942248:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828099.942265:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280719616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828099.942279:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828099.942284:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828099.942286:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828099.942290:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2649 00000100:00080000:3.0:1787828099.942298:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828099.942303:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828099.942309:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828099.942320:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828099.942331:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280719744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828099.942344:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828099.942349:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828099.942351:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828099.942358:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2649 00000100:00080000:3.0:1787828099.942364:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828099.942370:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828099.942376:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828099.942391:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280719872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828099.942435:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828099/real 1787828099] req@ffff8ab3d0a40000 x1874671280719360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828099.942466:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280719360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828099.942481:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828099.942486:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828099.942492:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280719360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828099.942523:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280719488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828099.942555:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828099.942572:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828099/real 1787828099] req@ffff8ab3d0a403c0 x1874671280719488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828099.942588:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280719488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828099.942616:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828099.942620:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828099.942624:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280719488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828099.942631:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280719616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828099.942650:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828099.942675:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828099/real 1787828099] req@ffff8ab3d0a421c0 x1874671280719616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828099.942690:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280719616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828099.942700:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828099.942704:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828099.942708:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280719616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828099.942715:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280719744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828099.942736:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828099.942748:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828099/real 1787828099] req@ffff8ab3d0a412c0 x1874671280719744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828099.942762:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280719744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828099.942783:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828099.942787:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828099.942792:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280719744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828099.942799:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280719872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828099.942818:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828099.942828:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828099/real 1787828099] req@ffff8ab3d0a43c00 x1874671280719872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828099.942842:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280719872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828099.942853:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828099.942856:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828099.942860:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280719872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828100.966122:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828101.990136:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828103.014170:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828104.038119:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828105.061991:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828105.061999:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828105.062003:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828105.062011:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2654 00000100:00080000:3.0:1787828105.062021:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828105.062026:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828105.062036:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828105.062065:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280720000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828105.062091:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828105.062095:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828105.062097:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828105.062102:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2654 00000100:00080000:3.0:1787828105.062107:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828105.062112:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828105.062120:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280720000:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787828105.062122:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828105.062141:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280720128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828105.062154:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828105.062158:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828105.062160:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828105.062165:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2654 00000100:00080000:3.0:1787828105.062169:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828105.062174:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828105.062178:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828105.062185:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828105.062198:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280720256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828105.062209:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828105.062212:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828105.062214:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828105.062218:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2654 00000100:00100000:1.0:1787828105.062221:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828105/real 1787828105] req@ffff8ab3d0a43c00 x1874671280720000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828105.062224:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828105.062232:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828105.062238:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828105.062241:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280720000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828105.062251:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828105.062255:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280720384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828105.062256:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828105.062260:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280720000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828105.062278:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280720128:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787828105.062289:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828105.062294:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828105.062297:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828105.062304:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828105.062304:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2654 00000100:00080000:3.0:1787828105.062311:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828105.062319:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828105.062320:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828105/real 1787828105] req@ffff8ab3d0a412c0 x1874671280720128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828105.062327:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828105.062332:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280720128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828105.062340:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828105.062345:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787828105.062345:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280720512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828105.062351:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280720128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828105.062360:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280720256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828105.062378:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828105.062387:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828105/real 1787828105] req@ffff8ab3d0a421c0 x1874671280720256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828105.062395:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280720256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828105.062401:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828105.062403:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828105.062405:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280720256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828105.062415:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280720384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828105.062426:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828105.062432:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828105/real 1787828105] req@ffff8ab3d0a403c0 x1874671280720384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828105.062439:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280720384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828105.062445:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828105.062448:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828105.062450:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280720384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828105.062453:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280720512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828105.062461:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828105.062466:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828105/real 1787828105] req@ffff8ab3d0a40000 x1874671280720512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828105.062473:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280720512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828105.062479:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828105.062480:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828105.062482:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280720512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828106.086102:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828107.110072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828108.134017:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828109.158112:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828110.181981:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828110.181992:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828110.181996:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828110.182007:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2659 00000100:00080000:3.0:1787828110.182018:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828110.182026:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828110.182066:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280720640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828110.182077:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828110.182114:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828110.182120:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828110.182123:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828110.182130:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2659 00000100:00080000:3.0:1787828110.182136:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828110.182144:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828110.182157:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828110.182159:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280720640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828110.182175:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280720768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828110.182188:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828110.182192:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828110.182194:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828110.182198:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2659 00000100:00080000:3.0:1787828110.182224:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828110.182230:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828110.182233:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828110.182241:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828110.182256:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280720896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828110.182265:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828110.182269:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828110.182271:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828110.182276:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2659 00000100:00080000:3.0:1787828110.182281:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828110.182286:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828110/real 1787828110] req@ffff8ab3d0a430c0 x1874671280720640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828110.182289:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828110.182300:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828110.182319:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280720640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828110.182322:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280721024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828110.182361:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828110.182362:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828110.182372:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828110.182375:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787828110.182376:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828110.182387:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280720640:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828110.182388:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2659 00000100:00080000:3.0:1787828110.182395:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828110.182402:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828110.182407:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828110.182417:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280721152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828110.182424:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280720768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828110.182455:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828110.182470:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828110/real 1787828110] req@ffff8ab3d0a43840 x1874671280720768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828110.182481:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280720768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828110.182488:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828110.182492:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828110.182495:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280720768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828110.182500:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280720896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828110.182513:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828110.182521:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828110/real 1787828110] req@ffff8ab3d0a43480 x1874671280720896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828110.182531:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280720896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828110.182538:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828110.182541:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828110.182543:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280720896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828110.182548:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280721024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828110.182562:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828110.182570:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828110/real 1787828110] req@ffff8ab3d0a41e00 x1874671280721024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828110.182580:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280721024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828110.182588:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828110.182591:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828110.182593:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280721024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828110.182608:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280721152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828110.182622:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828110.182631:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828110/real 1787828110] req@ffff8ab3d0a430c0 x1874671280721152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828110.182641:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280721152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828110.182649:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828110.182651:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828110.182653:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280721152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828111.206147:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828112.230067:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828113.254174:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828114.278011:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828115.302014:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828115.302037:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828115.302042:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828115.302054:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2664 00000100:00080000:3.0:1787828115.302066:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828115.302074:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828115.302106:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828115.302130:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280721280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828115.302171:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828115.302174:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828115.302176:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828115.302180:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2664 00000100:00080000:3.0:1787828115.302183:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828115.302188:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828115.302198:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828115.302228:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280721408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828115.302239:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828115.302245:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828115.302247:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280721280:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828115.302247:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828115.302256:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2664 00000100:00080000:3.0:1787828115.302263:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828115.302266:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828115.302270:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828115.302279:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280721536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828115.302287:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828115.302290:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828115.302291:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828115.302295:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2664 00000100:00080000:3.0:1787828115.302299:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828115.302302:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828115.302307:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828115.302318:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280721664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828115.302326:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828115.302329:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828115.302331:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828115.302334:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2664 00000100:00080000:3.0:1787828115.302354:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828115.302361:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828115.302365:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828115.302373:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828115.302403:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280721792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828115.302458:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828115/real 1787828115] req@ffff8ab3d0a40000 x1874671280721280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828115.302478:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280721280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828115.302491:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828115.302496:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828115.302502:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280721280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828115.302532:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280721408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828115.302563:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828115.302577:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828115/real 1787828115] req@ffff8ab3d0a403c0 x1874671280721408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828115.302588:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280721408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828115.302596:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828115.302599:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828115.302601:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280721408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828115.302607:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280721536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828115.302622:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828115.302630:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828115/real 1787828115] req@ffff8ab3d0a421c0 x1874671280721536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828115.302639:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280721536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828115.302646:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828115.302648:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828115.302651:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280721536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828115.302674:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280721664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828115.302693:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828115.302703:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828115/real 1787828115] req@ffff8ab3d0a412c0 x1874671280721664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828115.302715:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280721664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828115.302725:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828115.302728:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828115.302731:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280721664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828115.302737:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280721792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828115.302752:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828115.302762:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828115/real 1787828115] req@ffff8ab3d0a43c00 x1874671280721792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828115.302799:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280721792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828115.302828:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828115.302831:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828115.302834:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280721792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828116.326169:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828117.350091:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828118.374145:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828119.398163:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828120.422047:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828120.422069:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828120.422079:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828120.422098:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2670 00000100:00080000:3.0:1787828120.422116:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787828120.422120:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828120.422131:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828120.422201:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280721920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828120.422243:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828120.422253:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828120.422256:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828120.422261:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2670 00000100:00080000:3.0:1787828120.422266:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828120.422273:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828120.422289:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828120.422307:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280722048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828120.422326:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280721920:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828120.422334:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828120.422346:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828120.422350:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828120.422360:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2670 00000100:00080000:3.0:1787828120.422367:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828120.422375:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828120.422386:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828120.422415:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280722176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828120.422436:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828120.422441:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828120.422445:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828120.422452:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2670 00000100:00080000:3.0:1787828120.422461:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828120.422466:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828120.422479:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828120.422494:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828120.422524:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280722304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828120.422544:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828120.422551:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828120.422555:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828120.422565:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2670 00000100:00080000:3.0:1787828120.422574:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828120.422583:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828120/real 1787828120] req@ffff8ab3d0a43c00 x1874671280721920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828120.422593:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828120.422616:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828120.422628:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280721920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828120.422647:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828120.422652:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280722432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828120.422659:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828120.422674:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280721920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828120.422723:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280722048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828120.422782:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828120.422801:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828120/real 1787828120] req@ffff8ab3d0a412c0 x1874671280722048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828120.422818:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280722048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828120.422830:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828120.422834:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828120.422839:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280722048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828120.422846:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280722176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828120.422866:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828120.422877:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828120/real 1787828120] req@ffff8ab3d0a421c0 x1874671280722176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828120.422926:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280722176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828120.422946:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828120.422950:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828120.422954:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280722176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828120.422962:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280722304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828120.422985:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828120.422998:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828120/real 1787828120] req@ffff8ab3d0a403c0 x1874671280722304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828120.423012:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280722304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828120.423025:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828120.423028:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828120.423032:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280722304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828120.423039:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280722432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828120.423057:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828120.423068:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828120/real 1787828120] req@ffff8ab3d0a40000 x1874671280722432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828120.423082:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280722432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828120.423094:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828120.423097:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828120.423101:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280722432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828121.446111:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828122.470125:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828123.494075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828124.518137:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828125.542041:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828125.542053:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828125.542059:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828125.542072:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2675 00000100:00080000:0.0:1787828125.542085:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828125.542094:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828125.542136:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828125.542149:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280722560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828125.542186:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828125.542191:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828125.542194:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828125.542200:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2675 00000100:00080000:0.0:1787828125.542224:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828125.542231:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828125.542247:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828125.542247:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280722560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828125.542298:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280722688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828125.542322:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828125.542327:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828125.542329:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828125.542335:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2675 00000100:00080000:0.0:1787828125.542377:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828125.542382:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828125.542387:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828125.542396:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828125.542402:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280722816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828125.542413:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828125.542416:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828125.542418:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828125.542422:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2675 00000100:00080000:0.0:1787828125.542425:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828125.542430:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828125.542436:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828125.542451:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280722944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828125.542474:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828125.542479:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828125/real 1787828125] req@ffff8ab3f075c000 x1874671280722560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828125.542481:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828125.542487:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828125.542499:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2675 00000100:00080000:0.0:1787828125.542507:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828125.542511:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828125.542517:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828125.542521:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280722560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828125.542534:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280723072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828125.542545:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828125.542554:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828125.542558:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280722560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828125.542589:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280722688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828125.542614:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828125.542638:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828125/real 1787828125] req@ffff8ab3f075e940 x1874671280722688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828125.542649:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280722688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828125.542656:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828125.542659:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828125.542662:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280722688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828125.542667:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280722816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828125.542680:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828125.542688:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828125/real 1787828125] req@ffff8ab3f075f840 x1874671280722816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828125.542718:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280722816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828125.542757:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828125.542761:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828125.542766:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280722816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828125.542773:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280722944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828125.542826:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828125.542841:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828125/real 1787828125] req@ffff8ab3f075de00 x1874671280722944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828125.542852:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280722944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828125.542861:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828125.542864:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828125.542866:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280722944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828125.542872:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280723072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828125.542889:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828125.542897:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828125/real 1787828125] req@ffff8ab3f075f0c0 x1874671280723072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828125.542906:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280723072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828125.542932:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828125.542935:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828125.542938:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280723072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828126.566103:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828127.590063:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828128.614199:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828129.638125:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828130.662028:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828130.662040:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828130.662046:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828130.662059:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2680 00000100:00080000:0.0:1787828130.662074:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828130.662085:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828130.662141:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280723200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828130.662194:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1787828130.662197:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828130.662206:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828130.662212:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828130.662220:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2680 00000100:00080000:0.0:1787828130.662227:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828130.662233:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828130.662250:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828130.662271:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280723328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828130.662290:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828130.662296:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828130.662296:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280723200:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828130.662304:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828130.662331:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2680 00000100:00080000:0.0:1787828130.662336:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828130.662340:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828130.662345:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828130.662359:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280723456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828130.662373:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828130.662378:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828130.662380:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828130.662387:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2680 00000100:00080000:0.0:1787828130.662422:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828130.662429:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828130.662435:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828130.662439:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828130.662460:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280723584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828130.662493:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828130.662497:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828130.662501:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828130.662524:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2680 00000100:00080000:0.0:1787828130.662531:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828130.662537:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828130/real 1787828130] req@ffff8ab3f075cf00 x1874671280723200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828130.662545:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828130.662567:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828130.662611:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280723200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828130.662612:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280723712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828130.662738:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828130.662751:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828130.662761:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280723200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828130.662803:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280723328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828130.662841:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828130.662859:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828130/real 1787828130] req@ffff8ab3f075cb40 x1874671280723328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828130.662876:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280723328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828130.662924:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828130.662929:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828130.662934:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280723328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828130.662943:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280723456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828130.662968:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828130.662982:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828130/real 1787828130] req@ffff8ab3f075ed00 x1874671280723456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828130.662998:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280723456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828130.663008:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828130.663013:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828130.663017:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280723456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828130.663024:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280723584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828130.663044:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828130.663055:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828130/real 1787828130] req@ffff8ab3f075f480 x1874671280723584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828130.663069:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280723584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828130.663082:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828130.663085:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828130.663089:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280723584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828130.663096:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280723712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828130.663120:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828130.663141:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828130/real 1787828130] req@ffff8ab3f075da40 x1874671280723712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828130.663164:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280723712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828130.663184:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828130.663191:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828130.663198:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280723712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828131.686063:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828132.710179:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828133.734136:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828134.758145:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828135.782085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828135.782099:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828135.782105:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828135.782119:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2685 00000100:00080000:3.0:1787828135.782132:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787828135.782140:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828135.782143:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828135.782198:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280723840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828135.782234:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828135.782239:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828135.782242:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828135.782247:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2685 00000100:00080000:3.0:1787828135.782252:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828135.782258:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828135.782271:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828135.782289:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280723968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828135.782306:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280723840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828135.782308:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828135.782325:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828135.782328:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828135.782335:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2685 00000100:00080000:3.0:1787828135.782338:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828135.782343:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828135.782350:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828135.782369:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280724096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828135.782381:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828135.782386:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828135.782389:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828135.782394:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2685 00000100:00080000:3.0:1787828135.782398:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828135.782405:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828135.782408:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828135.782418:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828135.782439:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280724224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828135.782461:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828135.782465:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828135.782469:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828135.782474:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828135/real 1787828135] req@ffff8ab3d0a430c0 x1874671280723840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828135.782484:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2685 00000100:00080000:3.0:1787828135.782512:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828135.782523:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280723840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828135.782529:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828135.782551:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787828135.782552:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828135.782577:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828135.782583:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280723840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828135.782600:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280724352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828135.782622:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280723968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828135.782683:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828135.782703:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828135/real 1787828135] req@ffff8ab3d0a41e00 x1874671280723968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828135.782719:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280723968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828135.782739:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828135.782743:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828135.782747:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280723968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828135.782755:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280724096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828135.782774:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828135.782785:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828135/real 1787828135] req@ffff8ab3d0a43480 x1874671280724096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828135.782798:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280724096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828135.782809:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828135.782812:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828135.782815:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280724096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828135.782822:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280724224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828135.782840:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828135.782851:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828135/real 1787828135] req@ffff8ab3d0a43840 x1874671280724224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828135.782864:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280724224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828135.782875:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828135.782878:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828135.782883:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280724224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828135.782938:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280724352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828135.782961:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828135.782974:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828135/real 1787828135] req@ffff8ab3d0a40000 x1874671280724352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828135.782988:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280724352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828135.783000:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828135.783004:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828135.783008:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280724352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828136.806060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828137.830178:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828138.854114:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828139.878111:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828140.901983:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828140.901994:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828140.901999:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1787828140.902001:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828140.902011:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2690 00000100:00080000:0.0:1787828140.902019:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828140.902025:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828140.902058:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280724480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828140.902079:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828140.902083:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828140.902085:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828140.902089:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2690 00000100:00080000:0.0:1787828140.902093:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828140.902097:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828140.902106:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828140.902111:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280724480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828140.902119:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280724608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828140.902143:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828140.902146:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828140.902148:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828140.902152:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2690 00000100:00080000:0.0:1787828140.902167:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828140.902171:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828140.902175:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828140.902176:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828140.902187:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280724736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828140.902195:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828140.902198:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828140.902200:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828140.902204:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2690 00000100:00080000:0.0:1787828140.902207:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828140.902211:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828140.902215:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828140.902218:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828140/real 1787828140] req@ffff8ab3f075e580 x1874671280724480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828140.902224:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280724864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828140.902233:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828140.902235:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280724480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828140.902236:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828140.902238:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828140.902242:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2690 00000100:00080000:0.0:1787828140.902245:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828140.902246:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828140.902249:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787828140.902249:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000008:00100000:0.0:1787828140.902253:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828140.902254:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280724480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828140.902261:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280724992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828140.902276:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280724608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828140.902297:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828140.902312:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828140/real 1787828140] req@ffff8ab3f075da40 x1874671280724608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828140.902324:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280724608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828140.902334:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828140.902337:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828140.902341:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280724608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828140.902347:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280724736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828140.902367:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828140.902375:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828140/real 1787828140] req@ffff8ab3f075f480 x1874671280724736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828140.902383:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280724736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828140.902389:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828140.902391:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828140.902393:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280724736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828140.902396:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280724864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828140.902407:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828140.902413:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828140/real 1787828140] req@ffff8ab3f075ed00 x1874671280724864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828140.902420:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280724864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828140.902427:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828140.902428:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828140.902430:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280724864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828140.902433:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280724992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828140.902443:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828140.902448:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828140/real 1787828140] req@ffff8ab3f075cb40 x1874671280724992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828140.902455:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280724992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828140.902461:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828140.902463:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828140.902465:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280724992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828141.926086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828142.950102:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828143.974081:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828144.998032:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828146.021995:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828146.022003:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828146.022006:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828146.022016:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2695 00000100:00080000:0.0:1787828146.022023:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828146.022031:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828146.022051:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828146.022067:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280725120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828146.022090:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828146.022093:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828146.022095:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828146.022098:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2695 00000100:00080000:0.0:1787828146.022100:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828146.022103:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828146.022111:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828146.022119:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280725248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828146.022124:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280725120:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828146.022133:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828146.022138:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828146.022139:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828146.022143:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2695 00000100:00080000:0.0:1787828146.022146:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828146.022148:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828146.022152:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828146.022162:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280725376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828146.022168:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828146.022170:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828146.022171:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828146.022174:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2695 00000100:00080000:0.0:1787828146.022177:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828146.022179:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828146.022182:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787828146.022186:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828146.022192:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280725504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828146.022200:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828146.022202:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828146.022204:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828146.022207:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2695 00000100:00080000:0.0:1787828146.022209:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828146.022212:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828146.022215:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828146.022225:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280725632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828146.022233:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828146/real 1787828146] req@ffff8ab3f075cf00 x1874671280725120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828146.022249:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280725120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828146.022257:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828146.022260:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828146.022264:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280725120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828146.022282:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280725248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828146.022296:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828146.022304:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828146/real 1787828146] req@ffff8ab3f075f0c0 x1874671280725248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828146.022312:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280725248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828146.022318:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828146.022319:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828146.022321:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280725248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828146.022325:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280725376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828146.022335:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828146.022340:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828146/real 1787828146] req@ffff8ab3f075de00 x1874671280725376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828146.022347:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280725376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828146.022352:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828146.022354:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828146.022356:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280725376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828146.022359:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280725504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828146.022369:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828146.022375:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828146/real 1787828146] req@ffff8ab3f075f840 x1874671280725504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828146.022382:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280725504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828146.022388:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828146.022390:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828146.022391:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280725504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828146.022395:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280725632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828146.022403:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828146.022409:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828146/real 1787828146] req@ffff8ab3f075e940 x1874671280725632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828146.022416:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280725632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828146.022421:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828146.022423:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828146.022425:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280725632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828147.046173:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828148.070179:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828149.094125:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828150.118146:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828151.142107:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828151.142120:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828151.142126:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828151.142142:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2700 00010000:00010000:0.0:1787828151.142146:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828151.142156:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828151.142170:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828151.142223:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280725760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828151.142283:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828151.142290:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828151.142293:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828151.142301:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2700 00000100:00080000:2.0:1787828151.142309:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828151.142318:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828151.142351:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828151.142356:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280725760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828151.142383:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280725888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828151.142403:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828151.142408:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828151.142411:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828151.142417:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2700 00000100:00080000:2.0:1787828151.142436:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828151.142443:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828151.142452:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828151.142481:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280726016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828151.142505:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828151.142509:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828151.142513:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828151.142541:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2700 00000100:00100000:1.0:1787828151.142563:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280725888:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787828151.142567:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828151.142581:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828151.142593:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828151.142635:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280726144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828151.142651:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828151.142655:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828151.142659:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828151.142665:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2700 00000100:00080000:2.0:1787828151.142671:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828151.142675:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280726016:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787828151.142682:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828151.142697:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828151.142717:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280726272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828151.142764:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280726144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828151.142797:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280726272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828152.166065:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828153.190090:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828154.214165:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787828154.214386:0:10942:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:1.0:1787828154.214454:0:10942:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828154.214474:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787828154.214552:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787828154.214574:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787828154.214596:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787828154.214623:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:2.0:1787828154.214626:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828151/real 1787828154] req@ffff8ab3f043b0c0 x1874671280725888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828154.214699:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280725888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828154.214729:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828154.214738:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828154.214747:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280725888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828154.214762:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828151/real 1787828154] req@ffff8ab3f0438780 x1874671280726016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828154.214788:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280726016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828154.214805:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828154.214813:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828154.214820:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280726016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828154.214833:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828151/real 1787828154] req@ffff8ab3f0438b40 x1874671280725760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828154.214870:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280725760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828154.214937:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828154.214945:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828154.214952:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280725760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828154.214997:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828151/real 1787828154] req@ffff8ab3f043a580 x1874671280726144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828154.215023:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280726144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828154.215045:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828154.215051:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828154.215058:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280726144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828154.215072:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828151/real 1787828154] req@ffff8ab3f043ad00 x1874671280726272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828154.215094:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280726272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828154.215113:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828154.215120:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828154.215125:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280726272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787828155.238165:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828156.262046:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828156.262058:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828156.262063:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828156.262074:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2705 00000100:00080000:3.0:1787828156.262086:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787828156.262096:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828156.262096:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828156.262145:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280726400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828156.262187:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828156.262194:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828156.262198:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828156.262207:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2705 00000100:00080000:3.0:1787828156.262215:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828156.262223:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828156.262235:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828156.262237:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280726400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828156.262258:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280726528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828156.262273:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828156.262277:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828156.262280:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828156.262285:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2705 00000100:00080000:3.0:1787828156.262289:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828156.262294:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828156.262302:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828156.262323:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280726656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828156.262334:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828156.262363:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828156.262369:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828156.262373:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828156.262382:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2705 00000100:00080000:3.0:1787828156.262389:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828156.262408:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828156/real 1787828156] req@ffff8ab3d0a403c0 x1874671280726400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828156.262408:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828156.262426:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828156.262443:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280726400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828156.262458:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280726784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828156.262461:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828156.262476:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787828156.262516:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828156.262518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280726400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828156.262529:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828156.262535:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828156.262545:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2705 00000100:00080000:3.0:1787828156.262552:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828156.262559:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828156.262567:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280726528:192.168.203.124@tcp:38:kworker.0 00000008:00100000:3.0:1787828156.262573:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828156.262601:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a403c0 x1874671280726912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828156.262625:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828156.262656:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828156/real 1787828156] req@ffff8ab3d0a421c0 x1874671280726528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828156.262722:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280726528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828156.262739:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828156.262745:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828156.262751:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280726528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828156.262764:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280726656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828156.262842:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828156.262865:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828156/real 1787828156] req@ffff8ab3d0a412c0 x1874671280726656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828156.262920:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280726656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828156.262938:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828156.262944:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828156.262950:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280726656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828156.262961:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280726784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828156.262997:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828156.263017:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828156/real 1787828156] req@ffff8ab3d0a43c00 x1874671280726784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828156.263048:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280726784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828156.263066:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828156.263071:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828156.263076:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280726784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828156.263105:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280726912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828156.263150:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828156.263170:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828156/real 1787828156] req@ffff8ab3d0a403c0 x1874671280726912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828156.263200:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a403c0 x1874671280726912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828156.263216:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828156.263222:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828156.263228:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280726912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787828157.286124:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828158.310126:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828159.334261:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828160.358157:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828161.382045:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828161.382058:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828161.382064:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828161.382077:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2711 00000100:00080000:2.0:1787828161.382107:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828161.382123:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787828161.382134:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828161.382173:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280727040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828161.382215:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828161.382220:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828161.382223:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828161.382227:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2711 00000100:00080000:2.0:1787828161.382232:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828161.382238:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828161.382251:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828161.382277:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280727040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828161.382278:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280727168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828161.382306:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828161.382311:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828161.382313:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828161.382319:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2711 00000100:00080000:2.0:1787828161.382323:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828161.382328:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828161.382336:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828161.382351:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280727296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828161.382365:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828161.382369:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828161.382372:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828161.382379:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828161.382387:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2711 00000100:00080000:2.0:1787828161.382396:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828161.382401:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828161.382421:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828161.382432:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828161/real 1787828161] req@ffff8ab3f043ad00 x1874671280727040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828161.382473:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280727424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828161.382492:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828161.382493:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280727040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828161.382510:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828161.382519:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828161.382531:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828161.382532:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2711 00000100:00080000:1.0:1787828161.382546:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787828161.382547:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828161.382557:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280727040:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828161.382564:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828161.382576:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828161.382595:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280727552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828161.382602:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280727168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828161.382654:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828161.382675:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828161/real 1787828161] req@ffff8ab3f043a580 x1874671280727168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828161.382691:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280727168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828161.382702:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828161.382707:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828161.382711:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280727168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828161.382720:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280727296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828161.382739:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828161.382750:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828161/real 1787828161] req@ffff8ab3f0438b40 x1874671280727296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828161.382765:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280727296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828161.382774:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828161.382778:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828161.382782:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280727296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828161.382788:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280727424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828161.382806:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828161.382817:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828161/real 1787828161] req@ffff8ab3f0438780 x1874671280727424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828161.382830:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280727424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828161.382842:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828161.382845:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828161.382849:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280727424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828161.382868:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280727552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828161.382946:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828161.382971:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828161/real 1787828161] req@ffff8ab3f043b0c0 x1874671280727552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828161.382991:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280727552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828161.383003:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828161.383007:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828161.383012:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280727552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828162.406058:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828163.430105:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828164.454099:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828165.478107:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828166.502070:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828166.502083:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828166.502107:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828166.502121:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2716 00000100:00080000:0.0:1787828166.502136:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787828166.502146:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828166.502150:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828166.502229:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280727680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828166.502273:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828166.502278:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828166.502280:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828166.502285:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2716 00000100:00080000:0.0:1787828166.502291:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828166.502297:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828166.502310:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828166.502364:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280727680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828166.502365:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280727808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828166.502393:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828166.502397:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828166.502400:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828166.502406:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2716 00000100:00080000:0.0:1787828166.502434:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828166.502462:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828166.502468:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828166.502488:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280727936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828166.502501:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828166.502506:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828166.502508:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828166.502531:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2716 00000800:00000100:1.0:1787828166.502540:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828166.502541:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828166.502554:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828166.502562:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828166.502579:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280728064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828166.502593:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828166.502598:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828166.502601:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828166.502608:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2716 00000100:00080000:0.0:1787828166.502612:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828166.502618:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828166/real 1787828166] req@ffff8ab3f075c000 x1874671280727680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828166.502622:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828166.502635:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828166.502656:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280728192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828166.502666:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280727680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828166.502690:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828166.502697:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828166.502704:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280727680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828166.502736:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280727808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828166.502767:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828166.502783:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828166/real 1787828166] req@ffff8ab3f075e940 x1874671280727808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828166.502799:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280727808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828166.502818:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828166.502822:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828166.502826:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280727808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828166.502834:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280727936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828166.502852:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828166.502882:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828166/real 1787828166] req@ffff8ab3f075f840 x1874671280727936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828166.502895:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280727936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828166.502905:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828166.502908:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828166.502911:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280727936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828166.502973:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280728064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828166.503031:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828166.503054:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828166/real 1787828166] req@ffff8ab3f075de00 x1874671280728064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828166.503078:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280728064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828166.503095:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828166.503099:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828166.503104:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280728064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828166.503131:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280728192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828166.503159:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828166.503173:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828166/real 1787828166] req@ffff8ab3f075f0c0 x1874671280728192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828166.503205:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280728192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828166.503231:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828166.503236:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828166.503243:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280728192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787828167.526058:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828168.550097:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828169.574049:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828170.598155:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828171.622048:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828171.622062:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828171.622068:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828171.622083:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2721 00000100:00080000:1.0:1787828171.622098:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828171.622109:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828171.622153:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280728320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828171.622204:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828171.622212:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828171.622216:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828171.622223:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2721 00000100:00080000:1.0:1787828171.622231:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828171.622240:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828171.622256:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828171.622284:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280728448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828171.622298:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828171.622304:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828171.622307:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828171.622314:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2721 00000100:00080000:1.0:1787828171.622320:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828171.622326:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828171.622336:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828171.622342:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280728320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828171.622357:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280728576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828171.622404:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828171.622410:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828171.622413:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828171.622420:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2721 00000100:00080000:1.0:1787828171.622426:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828171.622433:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828171.622442:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828171.622459:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280728704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828171.622472:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828171.622479:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828171.622489:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828171.622490:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:1.0:1787828171.622508:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2721 00000100:00080000:1.0:1787828171.622517:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828171.622524:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828171.622533:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828171.622557:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280728832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828171.622601:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828171/real 1787828171] req@ffff8ab3d22fc780 x1874671280728320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828171.622643:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280728320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828171.622658:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828171.622666:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828171.622672:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280728320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828171.622687:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280728448:192.168.203.124@tcp:38:kworker.0 00010000:00010000:0.0:1787828171.622725:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1787828171.622762:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828171.622782:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828171/real 1787828171] req@ffff8ab3d22fe1c0 x1874671280728448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828171.622815:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280728448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828171.622824:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828171.622830:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828171.622835:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280728448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828171.622883:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280728576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828171.622997:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828171.623017:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828171/real 1787828171] req@ffff8ab3d22fde00 x1874671280728576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828171.623038:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280728576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828171.623060:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828171.623065:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828171.623070:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280728576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828171.623080:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280728704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828171.623110:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828171.623127:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828171/real 1787828171] req@ffff8ab3d22fd2c0 x1874671280728704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828171.623146:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280728704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828171.623161:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828171.623166:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828171.623171:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280728704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828171.623182:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280728832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828171.623214:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828171.623232:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828171/real 1787828171] req@ffff8ab3d22fed00 x1874671280728832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828171.623252:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280728832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828171.623264:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828171.623267:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828171.623271:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280728832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828172.646148:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828173.670107:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828174.694196:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828175.718128:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828176.742065:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828176.742077:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828176.742084:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828176.742102:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2726 00000100:00080000:0.0:1787828176.742119:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828176.742153:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828176.742204:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828176.742217:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280728960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828176.742275:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828176.742284:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828176.742289:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828176.742299:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2726 00000100:00080000:0.0:1787828176.742307:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828176.742316:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280728960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828176.742319:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828176.742366:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828176.742395:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280729088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828176.742412:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828176.742417:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828176.742421:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828176.742426:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2726 00000800:00000100:2.0:1787828176.742434:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828176.742436:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828176.742458:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828176.742467:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828176.742487:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280729216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828176.742501:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828176.742510:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828176/real 1787828176] req@ffff8ab3f075cf00 x1874671280728960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828176.742511:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828176.742518:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828176.742533:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2726 00000100:00080000:0.0:1787828176.742542:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828176.742549:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828176.742557:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280728960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828176.742560:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:2.0:1787828176.742582:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828176.742584:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280729344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828176.742594:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828176.742610:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280728960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828176.742618:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828176.742625:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828176.742628:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828176.742637:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2726 00000100:00080000:0.0:1787828176.742641:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828176.742647:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828176.742654:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828176.742655:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280729088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828176.742676:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280729472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828176.742698:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828176.742723:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828176/real 1787828176] req@ffff8ab3f075cb40 x1874671280729088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828176.742740:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280729088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828176.742751:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828176.742755:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828176.742760:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280729088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828176.742768:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280729216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828176.742787:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828176.742798:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828176/real 1787828176] req@ffff8ab3f075ed00 x1874671280729216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828176.742812:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280729216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828176.742821:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828176.742825:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828176.742828:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280729216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828176.742835:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280729344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828176.742853:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828176.742864:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828176/real 1787828176] req@ffff8ab3f075f480 x1874671280729344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828176.742877:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280729344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828176.742945:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828176.742949:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828176.742953:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280729344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828176.742974:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280729472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828176.742999:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828176.743013:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828176/real 1787828176] req@ffff8ab3f075da40 x1874671280729472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828176.743028:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280729472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828176.743040:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828176.743044:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828176.743048:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280729472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828177.766192:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828178.790093:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828179.814092:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828180.838172:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828181.861997:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828181.862010:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828181.862015:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828181.862026:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2731 00000100:00080000:0.0:1787828181.862037:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828181.862063:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828181.862124:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280729600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828181.862130:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828181.862179:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828181.862187:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828181.862191:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828181.862199:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2731 00000100:00080000:0.0:1787828181.862206:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828181.862215:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828181.862231:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828181.862243:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280729600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828181.862258:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280729728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828181.862277:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828181.862284:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828181.862287:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828181.862294:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2731 00000100:00080000:0.0:1787828181.862300:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828181.862333:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828181.862342:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828181.862375:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280729856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828181.862379:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828181.862386:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828181.862390:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828181.862392:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828181.862396:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2731 00000100:00080000:0.0:1787828181.862400:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828181.862405:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828181.862410:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828181.862421:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280729984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828181.862445:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828181/real 1787828181] req@ffff8ab3f075e580 x1874671280729600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828181.862451:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828181.862458:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828181.862462:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828181.862468:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2731 00000100:00080000:0.0:1787828181.862474:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828181.862477:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280729600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828181.862481:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828181.862490:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:2.0:1787828181.862497:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828181.862506:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828181.862507:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280730112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828181.862514:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280729600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828181.862552:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280729728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828181.862596:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828181.862628:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828181/real 1787828181] req@ffff8ab3f075da40 x1874671280729728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828181.862638:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280729728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828181.862645:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828181.862648:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828181.862650:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280729728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828181.862655:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280729856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828181.862669:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828181.862676:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828181/real 1787828181] req@ffff8ab3f075f480 x1874671280729856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828181.862685:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280729856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828181.862691:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828181.862693:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828181.862695:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280729856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828181.862699:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280729984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828181.862711:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828181.862717:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828181/real 1787828181] req@ffff8ab3f075ed00 x1874671280729984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828181.862725:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280729984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828181.862738:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828181.862741:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828181.862743:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280729984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828181.862747:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280730112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828181.862758:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828181.862764:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828181/real 1787828181] req@ffff8ab3f075cb40 x1874671280730112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828181.862772:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280730112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828181.862779:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828181.862781:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828181.862783:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280730112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828182.886088:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828183.910131:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828184.934129:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828185.958062:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828186.982059:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828186.982069:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828186.982074:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828186.982087:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2736 00010000:00010000:2.0:1787828186.982095:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828186.982098:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828186.982107:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828186.982171:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280730240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828186.982202:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828186.982205:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828186.982207:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828186.982211:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2736 00000100:00080000:0.0:1787828186.982215:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828186.982219:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828186.982232:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280730240:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787828186.982238:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828186.982254:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280730368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828186.982263:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828186.982266:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828186.982279:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828186.982283:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2736 00000100:00080000:0.0:1787828186.982308:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828186.982323:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828186.982327:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828186.982332:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828186.982341:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280730496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828186.982350:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828186.982352:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828186.982354:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828186.982357:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2736 00000100:00080000:0.0:1787828186.982359:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828186.982362:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828186.982366:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828186.982376:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280730624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828186.982384:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828186.982388:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828186/real 1787828186] req@ffff8ab3f075cf00 x1874671280730240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828186.982389:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828186.982392:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828186.982400:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2736 00000100:00080000:0.0:1787828186.982405:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828186.982409:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828186.982412:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828186.982417:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280730240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828186.982424:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280730752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828186.982453:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828186.982468:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828186.982472:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280730240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828186.982505:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280730368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828186.982534:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828186.982543:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828186/real 1787828186] req@ffff8ab3f075f0c0 x1874671280730368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828186.982552:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280730368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828186.982558:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828186.982561:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828186.982563:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280730368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828186.982567:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280730496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828186.982577:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828186.982583:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828186/real 1787828186] req@ffff8ab3f075de00 x1874671280730496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828186.982601:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280730496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828186.982606:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828186.982608:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828186.982611:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280730496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828186.982615:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280730624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828186.982626:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828186.982631:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828186/real 1787828186] req@ffff8ab3f075f840 x1874671280730624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828186.982655:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280730624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828186.982663:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828186.982665:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828186.982666:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280730624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828186.982670:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280730752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828186.982681:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828186.982687:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828186/real 1787828186] req@ffff8ab3f075e940 x1874671280730752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828186.982695:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280730752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828186.982701:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828186.982703:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828186.982705:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280730752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828188.006003:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828189.030037:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828190.054158:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828191.078034:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828192.102005:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828192.102013:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828192.102017:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828192.102026:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2741 00010000:00010000:3.0:1787828192.102032:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828192.102035:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828192.102042:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828192.102077:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280730880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828192.102097:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828192.102101:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828192.102103:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828192.102107:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2741 00000100:00080000:1.0:1787828192.102111:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828192.102115:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828192.102124:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828192.102136:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280731008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828192.102143:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280730880:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787828192.102148:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828192.102151:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828192.102152:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828192.102156:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2741 00000100:00080000:1.0:1787828192.102160:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828192.102164:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828192.102168:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828192.102177:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280731136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828192.102186:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828192.102189:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828192.102190:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828192.102193:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2741 00000100:00080000:1.0:1787828192.102196:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828192.102200:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787828192.102201:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:1.0:1787828192.102205:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828192.102215:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280731264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828192.102223:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828192.102227:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828192.102228:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828192.102250:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2741 00000100:00080000:1.0:1787828192.102254:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828192.102258:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828192.102260:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828192/real 1787828192] req@ffff8ab3d22fed00 x1874671280730880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828192.102263:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828192.102273:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280731392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828192.102278:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280730880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828192.102289:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828192.102293:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828192.102297:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280730880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828192.102322:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280731008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828192.102340:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828192.102351:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828192/real 1787828192] req@ffff8ab3d22fd2c0 x1874671280731008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828192.102359:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280731008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828192.102365:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828192.102367:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828192.102369:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280731008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828192.102373:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280731136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828192.102383:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828192.102388:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828192/real 1787828192] req@ffff8ab3d22fde00 x1874671280731136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828192.102401:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280731136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828192.102406:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828192.102407:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828192.102409:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280731136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828192.102412:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280731264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828192.102422:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828192.102427:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828192/real 1787828192] req@ffff8ab3d22fe1c0 x1874671280731264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828192.102435:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280731264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828192.102441:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828192.102442:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828192.102444:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280731264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828192.102447:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280731392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828192.102457:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828192.102462:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828192/real 1787828192] req@ffff8ab3d22fc780 x1874671280731392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828192.102469:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280731392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828192.102479:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828192.102481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828192.102483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280731392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828193.126021:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828194.150054:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828195.174062:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828196.198011:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828197.222004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828197.222019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828197.222025:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828197.222039:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2746 00000100:00080000:3.0:1787828197.222053:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828197.222062:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828197.222098:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828197.222109:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280731520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828197.222136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828197.222139:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828197.222141:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828197.222144:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2746 00000100:00080000:3.0:1787828197.222147:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828197.222150:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828197.222160:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828197.222172:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280731520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828197.222173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280731648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828197.222194:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828197.222211:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828197.222213:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828197.222220:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2746 00000100:00080000:3.0:1787828197.222225:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828197.222230:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828197.222235:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828197.222247:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280731776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828197.222255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828197.222262:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828197.222266:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828197.222269:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828197.222274:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2746 00000100:00080000:3.0:1787828197.222280:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828197.222284:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828197.222288:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828197.222299:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280731904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828197.222306:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828197.222310:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828197.222312:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828197.222316:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828197/real 1787828197] req@ffff8ab3d0a40000 x1874671280731520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828197.222316:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2746 00000100:00080000:3.0:1787828197.222326:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828197.222339:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828197.222354:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828197.222362:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280731520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828197.222368:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280732032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828197.222410:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828197.222416:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828197.222420:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280731520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828197.222443:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280731648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828197.222459:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828197.222468:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828197/real 1787828197] req@ffff8ab3d0a43840 x1874671280731648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828197.222476:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280731648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828197.222486:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828197.222488:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828197.222490:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280731648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828197.222494:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280731776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828197.222504:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828197.222510:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828197/real 1787828197] req@ffff8ab3d0a43480 x1874671280731776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828197.222517:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280731776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828197.222522:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828197.222524:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828197.222526:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280731776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828197.222529:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280731904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828197.222540:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828197.222546:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828197/real 1787828197] req@ffff8ab3d0a41e00 x1874671280731904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828197.222553:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280731904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828197.222559:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828197.222561:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828197.222563:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280731904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828197.222568:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280732032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828197.222577:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828197.222582:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828197/real 1787828197] req@ffff8ab3d0a430c0 x1874671280732032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828197.222589:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280732032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828197.222595:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828197.222597:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828197.222598:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280732032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828198.246032:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828199.270005:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828200.294042:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828201.318037:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828202.342011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828202.342021:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1787828202.342022:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828202.342026:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828202.342039:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2751 00000100:00080000:2.0:1787828202.342050:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828202.342057:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828202.342099:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280732160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828202.342125:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828202.342128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828202.342129:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828202.342133:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2751 00000100:00080000:2.0:1787828202.342136:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828202.342139:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828202.342149:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828202.342169:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280732160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828202.342176:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280732288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828202.342186:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828202.342189:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828202.342190:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828202.342194:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2751 00000100:00080000:2.0:1787828202.342197:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828202.342200:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828202.342204:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828202.342215:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280732416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828202.342222:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828202.342224:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828202.342226:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828202.342229:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2751 00000100:00080000:2.0:1787828202.342231:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828202.342234:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828202.342237:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828202.342243:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828202.342247:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280732544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828202.342255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828202.342258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828202.342260:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828202.342263:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2751 00000100:00080000:2.0:1787828202.342265:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828202.342268:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828202.342271:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828202.342279:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280732672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828202.342296:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828202/real 1787828202] req@ffff8ab3f0438f00 x1874671280732160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828202.342312:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280732160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828202.342321:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828202.342325:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828202.342328:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280732160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828202.342350:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280732288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828202.342365:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828202.342374:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828202/real 1787828202] req@ffff8ab3f043b0c0 x1874671280732288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828202.342385:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280732288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828202.342394:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828202.342398:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828202.342403:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280732288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828202.342410:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280732416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828202.342430:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828202.342441:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828202/real 1787828202] req@ffff8ab3f0438780 x1874671280732416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828202.342449:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280732416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828202.342455:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828202.342457:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828202.342460:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280732416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828202.342463:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280732544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828202.342477:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828202.342483:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828202/real 1787828202] req@ffff8ab3f0438b40 x1874671280732544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828202.342491:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280732544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828202.342497:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828202.342499:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828202.342501:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280732544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828202.342505:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280732672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828202.342525:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828202.342532:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828202/real 1787828202] req@ffff8ab3f043a580 x1874671280732672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828202.342539:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280732672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828202.342546:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828202.342548:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828202.342550:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280732672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828203.366105:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828204.390037:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828205.414051:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828206.438031:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828207.462000:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828207.462012:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1787828207.462015:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828207.462017:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828207.462028:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2757 00000100:00080000:0.0:1787828207.462038:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828207.462045:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828207.462081:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280732800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828207.462106:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828207.462110:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828207.462112:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828207.462117:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2757 00000100:00080000:0.0:1787828207.462122:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828207.462145:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828207.462156:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828207.462158:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280732800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828207.462174:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280732928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828207.462187:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828207.462192:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828207.462194:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828207.462198:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2757 00000100:00080000:0.0:1787828207.462204:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828207.462209:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828207.462214:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828207.462218:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828207.462233:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280733056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828207.462243:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828207.462246:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828207.462248:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828207.462256:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2757 00000100:00100000:1.0:1787828207.462256:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828207/real 1787828207] req@ffff8ab3f075c000 x1874671280732800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828207.462263:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828207.462272:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828207.462277:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280732800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828207.462278:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787828207.462291:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828207.462296:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280733184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828207.462299:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828207.462308:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280732800:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828207.462313:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828207.462318:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828207.462320:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828207.462326:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2757 00000100:00080000:0.0:1787828207.462329:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828207.462333:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828207.462339:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828207.462339:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280732928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828207.462350:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280733312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828207.462381:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828207.462396:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828207/real 1787828207] req@ffff8ab3f075e940 x1874671280732928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828207.462405:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280732928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828207.462416:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828207.462418:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828207.462421:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280732928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828207.462425:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280733056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828207.462438:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828207.462444:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828207/real 1787828207] req@ffff8ab3f075f840 x1874671280733056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828207.462451:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280733056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828207.462456:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828207.462458:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828207.462460:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280733056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828207.462463:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280733184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828207.462473:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828207.462479:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828207/real 1787828207] req@ffff8ab3f075de00 x1874671280733184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828207.462486:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280733184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828207.462492:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828207.462493:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828207.462496:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280733184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828207.462506:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280733312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828207.462516:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828207.462521:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828207/real 1787828207] req@ffff8ab3f075f0c0 x1874671280733312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828207.462528:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280733312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828207.462534:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828207.462535:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828207.462537:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280733312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828208.486037:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828209.510041:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828210.534042:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828211.558045:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828212.581999:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828212.582007:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828212.582011:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828212.582024:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2762 00000100:00080000:0.0:1787828212.582051:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828212.582063:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828212.582082:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828212.582147:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280733440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828212.582174:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828212.582177:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828212.582179:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828212.582182:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2762 00000100:00080000:0.0:1787828212.582185:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828212.582189:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828212.582198:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828212.582210:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280733440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828212.582211:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280733568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828212.582224:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828212.582227:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828212.582228:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828212.582231:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2762 00000100:00080000:0.0:1787828212.582233:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828212.582236:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828212.582239:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828212.582247:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280733696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828212.582259:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828212.582261:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828212.582262:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828212.582264:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2762 00000100:00080000:0.0:1787828212.582267:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828212.582271:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828212.582274:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828212.582279:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828212.582287:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280733824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828212.582293:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828212.582296:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828212.582297:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828212.582301:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2762 00000100:00080000:0.0:1787828212.582303:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828212.582306:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828212.582309:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828212.582318:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280733952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828212.582318:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828212/real 1787828212] req@ffff8ab3f075cf00 x1874671280733440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828212.582356:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280733440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828212.582366:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828212.582369:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828212.582372:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280733440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828212.582392:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280733568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828212.582410:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828212.582420:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828212/real 1787828212] req@ffff8ab3f075cb40 x1874671280733568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828212.582438:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280733568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828212.582444:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828212.582446:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828212.582449:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280733568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828212.582453:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280733696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828212.582465:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828212.582471:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828212/real 1787828212] req@ffff8ab3f075ed00 x1874671280733696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828212.582479:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280733696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828212.582485:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828212.582488:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828212.582492:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280733696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828212.582498:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280733824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828212.582518:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828212.582529:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828212/real 1787828212] req@ffff8ab3f075f480 x1874671280733824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828212.582542:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280733824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828212.582550:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828212.582552:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828212.582554:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280733824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828212.582559:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280733952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828212.582574:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828212.582580:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828212/real 1787828212] req@ffff8ab3f075da40 x1874671280733952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828212.582588:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280733952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828212.582594:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828212.582596:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828212.582598:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280733952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828213.606072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828214.630156:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828215.654092:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828216.678032:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828217.702017:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828217.702025:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828217.702030:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828217.702039:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2767 00000100:00080000:2.0:1787828217.702050:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1787828217.702070:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828217.702071:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828217.702111:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280734080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828217.702134:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828217.702137:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828217.702138:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828217.702141:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2767 00000100:00080000:2.0:1787828217.702144:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828217.702146:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828217.702155:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828217.702165:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280734208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828217.702174:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828217.702176:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828217.702177:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828217.702180:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2767 00000100:00080000:2.0:1787828217.702183:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828217.702191:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828217.702195:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280734080:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787828217.702195:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828217.702209:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280734336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828217.702216:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828217.702218:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828217.702220:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828217.702224:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2767 00000100:00080000:2.0:1787828217.702229:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828217.702232:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828217.702237:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828217.702246:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280734464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828217.702259:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828217.702261:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828217.702262:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828217.702266:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2767 00000100:00080000:2.0:1787828217.702272:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828217.702275:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828217.702279:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828217.702287:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280734592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828217.702350:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280734208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828217.702385:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280734336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828217.702403:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280734464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828217.702422:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280734592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828218.726091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828219.750067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828220.774115:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828220.774285:0:10944:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787828220.774333:0:10944:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828220.774357:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828220.774404:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828220.774417:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828220.774428:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828220.774439:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787828220.774468:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828217/real 1787828220] req@ffff8ab3f0439680 x1874671280734208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828220.774517:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280734208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828220.774532:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828220.774540:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828220.774548:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280734208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828220.774560:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828217/real 1787828220] req@ffff8ab3f043bc00 x1874671280734336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828220.774579:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280734336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828220.774589:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828220.774593:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828220.774597:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280734336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828220.774605:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828217/real 1787828220] req@ffff8ab3f04383c0 x1874671280734464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828220.774618:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280734464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828220.774631:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828220.774635:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828220.774639:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280734464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828220.774647:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828217/real 1787828220] req@ffff8ab3f04392c0 x1874671280734592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828220.774662:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280734592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828220.774675:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828220.774679:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828220.774692:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280734592:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828220.774701:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828217/real 1787828220] req@ffff8ab3f043ad00 x1874671280734080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828220.774714:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280734080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828220.774724:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828220.774728:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828220.774732:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280734080:192.168.203.124@tcp:250:kworker.0 00010000:00010000:3.0:1787828221.798063:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828222.822085:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828222.822095:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828222.822098:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1787828222.822106:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828222.822108:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2772 00000100:00080000:0.0:1787828222.822120:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828222.822127:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828222.822160:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280734720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828222.822190:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828222.822193:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828222.822195:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828222.822199:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2772 00000100:00080000:0.0:1787828222.822204:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828222.822210:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828222.822220:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828222.822234:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280734848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828222.822244:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280734720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828222.822248:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828222.822258:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828222.822261:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828222.822266:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2772 00000100:00080000:0.0:1787828222.822269:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828222.822272:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828222.822276:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828222.822286:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280734976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828222.822293:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828222.822295:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828222.822296:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828222.822299:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2772 00000100:00080000:0.0:1787828222.822301:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828222.822304:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828222.822307:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828222.822314:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280735104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828222.822320:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828222.822323:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828222.822324:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828222.822326:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2772 00000100:00080000:0.0:1787828222.822339:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828222.822343:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828222.822350:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787828222.822355:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828222.822370:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280735232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828222.822429:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828222/real 1787828222] req@ffff8ab3f075e580 x1874671280734720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828222.822455:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280734720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828222.822471:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828222.822476:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828222.822481:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280734720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828222.822518:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280734848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828222.822546:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828222.822561:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828222/real 1787828222] req@ffff8ab3f075da40 x1874671280734848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828222.822575:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280734848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828222.822584:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828222.822588:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828222.822592:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280734848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828222.822601:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280734976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828222.822626:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828222.822638:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828222/real 1787828222] req@ffff8ab3f075f480 x1874671280734976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828222.822649:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280734976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828222.822658:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828222.822661:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828222.822665:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280734976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828222.822671:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280735104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828222.822694:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828222.822708:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828222/real 1787828222] req@ffff8ab3f075ed00 x1874671280735104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828222.822717:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280735104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828222.822728:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828222.822732:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828222.822735:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280735104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828222.822742:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280735232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828222.822764:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828222.822776:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828222/real 1787828222] req@ffff8ab3f075cb40 x1874671280735232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828222.822787:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280735232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828222.822796:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828222.822800:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828222.822804:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280735232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828223.846143:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828224.870026:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828225.894048:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828226.918045:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828227.941978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828227.941999:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828227.942002:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828227.942010:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2777 00000100:00080000:2.0:1787828227.942020:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828227.942026:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828227.942038:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828227.942060:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280735360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828227.942085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828227.942088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828227.942089:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828227.942093:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2777 00000100:00080000:2.0:1787828227.942096:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828227.942099:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828227.942107:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828227.942117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280735488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828227.942122:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280735360:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828227.942127:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828227.942132:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828227.942134:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828227.942137:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2777 00000100:00080000:2.0:1787828227.942139:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828227.942142:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828227.942145:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828227.942153:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280735616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828227.942159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828227.942162:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828227.942163:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828227.942166:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2777 00000100:00080000:2.0:1787828227.942186:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828227.942189:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828227.942192:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828227.942197:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828227.942201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280735744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828227.942210:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828227.942212:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828227.942214:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828227.942217:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2777 00000100:00080000:2.0:1787828227.942219:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828227.942222:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828227.942226:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828227.942233:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280735872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828227.942255:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828227/real 1787828227] req@ffff8ab3f0439a40 x1874671280735360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828227.942270:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280735360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828227.942283:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828227.942286:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828227.942289:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280735360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828227.942307:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280735488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828227.942323:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828227.942332:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828227/real 1787828227] req@ffff8ab3f043a1c0 x1874671280735488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828227.942339:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280735488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828227.942345:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828227.942347:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828227.942349:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280735488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828227.942353:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280735616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828227.942362:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828227.942368:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828227/real 1787828227] req@ffff8ab3f043a940 x1874671280735616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828227.942375:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280735616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828227.942380:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828227.942382:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828227.942384:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280735616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828227.942388:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280735744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828227.942398:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828227.942403:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828227/real 1787828227] req@ffff8ab3f043b480 x1874671280735744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828227.942410:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280735744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828227.942416:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828227.942418:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828227.942419:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280735744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828227.942423:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280735872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828227.942431:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828227.942437:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828227/real 1787828227] req@ffff8ab3f0439e00 x1874671280735872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828227.942444:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280735872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828227.942449:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828227.942451:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828227.942453:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280735872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828228.966006:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828229.990038:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828231.014026:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828232.038140:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828233.062009:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828233.062017:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1787828233.062017:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828233.062023:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828233.062031:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2782 00000100:00080000:0.0:1787828233.062039:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828233.062045:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828233.062078:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280736000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828233.062100:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828233.062102:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828233.062104:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828233.062106:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2782 00000100:00080000:0.0:1787828233.062109:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828233.062112:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828233.062119:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828233.062131:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280736128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828233.062132:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280736000:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828233.062145:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828233.062148:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828233.062149:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828233.062152:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2782 00000100:00080000:0.0:1787828233.062154:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828233.062156:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828233.062161:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828233.062171:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280736256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828233.062177:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828233.062180:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828233.062181:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828233.062184:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2782 00000100:00080000:0.0:1787828233.062186:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828233.062188:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828233.062192:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828233.062192:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828233.062203:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280736384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828233.062211:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828233.062213:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828233.062214:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828233.062217:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2782 00000100:00080000:0.0:1787828233.062219:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828233.062221:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828233.062224:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828233.062233:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280736512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828233.062239:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828233/real 1787828233] req@ffff8ab3f075cf00 x1874671280736000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828233.062256:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280736000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828233.062264:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828233.062268:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828233.062270:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280736000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828233.062288:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280736128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828233.062302:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828233.062311:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828233/real 1787828233] req@ffff8ab3f075f0c0 x1874671280736128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828233.062319:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280736128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828233.062325:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828233.062327:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828233.062328:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280736128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828233.062332:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280736256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828233.062343:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828233.062348:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828233/real 1787828233] req@ffff8ab3f075de00 x1874671280736256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828233.062358:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280736256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828233.062382:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828233.062386:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828233.062389:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280736256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828233.062396:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280736384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828233.062417:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828233.062426:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828233/real 1787828233] req@ffff8ab3f075f840 x1874671280736384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828233.062434:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280736384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828233.062441:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828233.062443:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828233.062445:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280736384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828233.062450:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280736512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828233.062462:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828233.062468:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828233/real 1787828233] req@ffff8ab3f075e940 x1874671280736512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828233.062475:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280736512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828233.062480:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828233.062482:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828233.062484:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280736512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828234.086139:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828235.110096:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828236.134110:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828237.158066:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828238.182069:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828238.182084:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828238.182091:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828238.182108:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2787 00000100:00080000:0.0:1787828238.182135:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828238.182148:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828238.182209:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280736640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828238.182223:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828238.182251:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828238.182267:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828238.182272:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828238.182282:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2787 00000100:00080000:0.0:1787828238.182291:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828238.182301:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828238.182321:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280736640:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787828238.182324:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828238.182364:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280736768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828238.182389:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828238.182396:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828238.182399:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828238.182408:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2787 00000100:00080000:0.0:1787828238.182432:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828238.182441:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828238.182452:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828238.182457:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828238.182490:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280736896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828238.182523:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828238.182533:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828238/real 1787828238] req@ffff8ab3f075c000 x1874671280736640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828238.182536:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828238.182546:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828238.182567:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2787 00000100:00080000:0.0:1787828238.182576:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828238.182582:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280736640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828238.182593:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787828238.182616:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828238.182617:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787828238.182630:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828238.182638:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280736640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828238.182659:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280737024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828238.182679:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280736768:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787828238.182692:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828238.182702:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828238.182707:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828238.182716:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2787 00000800:00000100:1.0:1787828238.182724:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828238.182727:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828238.182739:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828238.182751:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828238.182753:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828238/real 1787828238] req@ffff8ab3f075e940 x1874671280736768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828238.182780:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280737152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828238.182783:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280736768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828238.182838:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828238.182845:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828238.182849:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280736768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828238.182861:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280736896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828238.182955:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828238.182976:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828238/real 1787828238] req@ffff8ab3f075f840 x1874671280736896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828238.182992:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280736896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828238.183004:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828238.183008:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828238.183012:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280736896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828238.183032:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280737024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828238.183059:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828238.183071:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828238/real 1787828238] req@ffff8ab3f075de00 x1874671280737024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828238.183086:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280737024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828238.183098:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828238.183101:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828238.183104:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280737024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828238.183111:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280737152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828238.183129:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828238.183140:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828238/real 1787828238] req@ffff8ab3f075f0c0 x1874671280737152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828238.183153:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280737152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828238.183166:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828238.183169:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828238.183173:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280737152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828239.206155:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828240.230113:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828241.254070:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828242.278178:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828243.302085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828243.302107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828243.302116:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828243.302164:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2792 00000100:00080000:0.0:1787828243.302189:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828243.302222:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828243.302234:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828243.302289:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280737280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828243.302348:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828243.302352:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828243.302355:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828243.302359:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2792 00000100:00080000:0.0:1787828243.302363:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828243.302367:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828243.302381:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828243.302398:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280737408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828243.302411:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828243.302417:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828243.302417:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280737280:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828243.302423:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828243.302435:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2792 00000100:00080000:0.0:1787828243.302439:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828243.302442:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828243.302448:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828243.302463:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280737536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828243.302474:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828243.302478:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828243.302480:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828243.302485:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2792 00000100:00080000:0.0:1787828243.302505:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828243.302510:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828243.302516:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828243.302523:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828243.302535:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280737664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828243.302547:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828243.302552:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828243.302554:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828243.302559:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2792 00000100:00080000:0.0:1787828243.302563:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828243.302567:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828243.302572:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828243.302583:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280737792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828243.302633:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828243/real 1787828243] req@ffff8ab3f075cf00 x1874671280737280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828243.302663:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280737280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828243.302677:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828243.302682:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828243.302687:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280737280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828243.302750:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280737408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828243.302783:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828243.302800:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828243/real 1787828243] req@ffff8ab3f075cb40 x1874671280737408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828243.302815:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280737408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828243.302826:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828243.302831:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828243.302834:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280737408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828243.302843:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280737536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828243.302864:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828243.302875:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828243/real 1787828243] req@ffff8ab3f075ed00 x1874671280737536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828243.302989:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280737536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828243.303000:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828243.303004:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828243.303008:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280737536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828243.303016:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280737664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828243.303046:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828243.303078:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828243/real 1787828243] req@ffff8ab3f075f480 x1874671280737664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828243.303093:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280737664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828243.303106:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828243.303110:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828243.303113:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280737664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828243.303121:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280737792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828243.303148:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828243.303169:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828243/real 1787828243] req@ffff8ab3f075da40 x1874671280737792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828243.303212:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280737792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828243.303230:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828243.303236:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828243.303243:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280737792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828244.326042:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828245.350140:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828246.374171:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828247.398105:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828248.422037:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828248.422053:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828248.422060:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828248.422078:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2798 00000100:00080000:0.0:1787828248.422095:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828248.422108:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828248.422146:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828248.422176:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280737920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828248.422218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828248.422223:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828248.422226:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828248.422231:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2798 00000100:00080000:0.0:1787828248.422236:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828248.422242:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828248.422259:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828248.422285:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280737920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828248.422286:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280738048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828248.422315:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828248.422320:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828248.422323:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828248.422330:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2798 00000100:00080000:0.0:1787828248.422335:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828248.422340:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828248.422347:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828248.422366:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280738176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828248.422388:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828248.422395:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828248.422399:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828248.422408:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828248.422418:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2798 00000100:00080000:0.0:1787828248.422431:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828248.422443:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828248.422457:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828248.422487:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280738304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828248.422496:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828248/real 1787828248] req@ffff8ab3f075e580 x1874671280737920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828248.422513:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828248.422532:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828248.422536:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828248.422544:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2798 00000100:00100000:1.0:1787828248.422552:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280737920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828248.422555:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828248.422571:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787828248.422582:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828248.422584:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828248.422601:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280738432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828248.422614:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828248.422630:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280737920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828248.422686:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280738048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828248.422717:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828248.422735:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828248/real 1787828248] req@ffff8ab3f075da40 x1874671280738048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828248.422761:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280738048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828248.422781:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828248.422786:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828248.422790:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280738048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828248.422798:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280738176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828248.422817:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828248.422828:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828248/real 1787828248] req@ffff8ab3f075f480 x1874671280738176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828248.422874:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280738176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828248.422925:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828248.422930:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828248.422934:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280738176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828248.422943:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280738304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828248.422969:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828248.422984:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828248/real 1787828248] req@ffff8ab3f075ed00 x1874671280738304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828248.422999:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280738304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828248.423012:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828248.423016:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828248.423021:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280738304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828248.423028:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280738432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828248.423055:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828248.423066:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828248/real 1787828248] req@ffff8ab3f075cb40 x1874671280738432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828248.423079:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280738432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828248.423091:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828248.423094:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828248.423098:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280738432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828249.446217:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828250.470037:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828251.494084:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828252.518040:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828253.542028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828253.542040:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828253.542073:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828253.542084:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2803 00000100:00080000:0.0:1787828253.542096:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828253.542125:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828253.542153:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828253.542173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280738560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828253.542203:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828253.542211:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828253.542230:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828253.542235:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2803 00000100:00080000:0.0:1787828253.542240:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828253.542246:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828253.542258:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828253.542271:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280738560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828253.542279:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280738688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828253.542295:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828253.542299:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828253.542301:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828253.542306:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2803 00000100:00080000:0.0:1787828253.542310:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828253.542314:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828253.542320:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828253.542333:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280738816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828253.542345:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828253.542349:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828253.542354:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828253.542355:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828253.542374:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2803 00000100:00080000:0.0:1787828253.542380:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828253.542385:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828253.542392:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828253.542406:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280738944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828253.542417:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828253.542422:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828253.542424:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828253.542430:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2803 00000100:00100000:1.0:1787828253.542435:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828253/real 1787828253] req@ffff8ab3f075cf00 x1874671280738560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828253.542439:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828253.542455:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828253.542467:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828253.542470:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280738560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828253.542487:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828253.542495:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828253.542496:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280739072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828253.542502:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280738560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828253.542543:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280738688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828253.542589:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828253.542622:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828253/real 1787828253] req@ffff8ab3f075f0c0 x1874671280738688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828253.542635:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280738688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828253.542644:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828253.542647:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828253.542668:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280738688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828253.542675:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280738816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828253.542697:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828253.542707:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828253/real 1787828253] req@ffff8ab3f075de00 x1874671280738816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828253.542719:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280738816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828253.542728:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828253.542731:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828253.542734:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280738816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828253.542740:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280738944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828253.542757:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828253.542766:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828253/real 1787828253] req@ffff8ab3f075f840 x1874671280738944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828253.542778:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280738944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828253.542789:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828253.542792:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828253.542796:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280738944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828253.542851:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280739072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828253.542872:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828253.542883:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828253/real 1787828253] req@ffff8ab3f075e940 x1874671280739072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828253.542921:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280739072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828253.542931:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828253.542934:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828253.542937:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280739072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828254.566113:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828255.590117:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828256.614079:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828257.638093:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828258.661979:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828258.661987:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828258.661991:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828258.662000:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2808 00000100:00080000:0.0:1787828258.662008:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828258.662014:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828258.662032:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828258.662054:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280739200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828258.662083:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828258.662087:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828258.662089:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828258.662093:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2808 00000100:00080000:0.0:1787828258.662098:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828258.662102:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828258.662110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828258.662112:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280739200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828258.662124:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280739328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828258.662134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828258.662137:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828258.662138:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828258.662141:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2808 00000100:00080000:0.0:1787828258.662143:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828258.662146:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828258.662149:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828258.662157:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280739456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828258.662163:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828258.662166:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828258.662167:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828258.662169:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2808 00000100:00080000:0.0:1787828258.662171:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828258.662177:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828258.662177:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828258.662182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828258.662190:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280739584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828258.662201:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828258.662204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828258.662205:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828258.662208:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2808 00000100:00080000:0.0:1787828258.662211:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828258.662214:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828258.662217:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828258.662222:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828258/real 1787828258] req@ffff8ab3f075c000 x1874671280739200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828258.662227:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280739712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828258.662251:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280739200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828258.662261:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828258.662264:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828258.662267:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280739200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828258.662285:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280739328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828258.662300:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828258.662309:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828258/real 1787828258] req@ffff8ab3f075e940 x1874671280739328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828258.662317:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280739328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828258.662322:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828258.662325:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828258.662327:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280739328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828258.662330:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280739456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828258.662340:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828258.662345:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828258/real 1787828258] req@ffff8ab3f075f840 x1874671280739456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828258.662353:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280739456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828258.662358:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828258.662360:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828258.662361:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280739456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828258.662365:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280739584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828258.662375:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828258.662381:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828258/real 1787828258] req@ffff8ab3f075de00 x1874671280739584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828258.662388:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280739584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828258.662394:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828258.662396:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828258.662398:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280739584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828258.662401:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280739712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828258.662410:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828258.662416:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828258/real 1787828258] req@ffff8ab3f075f0c0 x1874671280739712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828258.662423:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280739712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828258.662429:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828258.662430:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828258.662432:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280739712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828259.686035:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828260.710059:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828261.734120:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828262.758025:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828263.781973:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828263.781986:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828263.781991:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828263.782004:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2813 00000100:00080000:0.0:1787828263.782018:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828263.782026:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828263.782066:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280739840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828263.782086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828263.782089:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828263.782091:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828263.782094:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2813 00000100:00080000:0.0:1787828263.782097:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828263.782100:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828263.782110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828263.782120:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280739968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828263.782135:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828263.782139:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828263.782142:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828263.782146:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2813 00000100:00080000:0.0:1787828263.782149:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828263.782164:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828263.782164:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828263.782166:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280739840:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787828263.782183:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828263.782200:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280740096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828263.782209:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828263.782213:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828263.782214:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828263.782218:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2813 00000100:00080000:0.0:1787828263.782220:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828263.782223:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828263.782227:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828263.782235:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280740224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828263.782242:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828263.782245:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828263.782247:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828263.782251:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2813 00000100:00080000:0.0:1787828263.782289:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828263.782292:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828263.782296:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787828263.782311:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828263.782312:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280740352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828263.782399:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828263/real 1787828263] req@ffff8ab3f075cf00 x1874671280739840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828263.782441:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280739840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828263.782459:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828263.782465:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828263.782471:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280739840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828263.782508:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280739968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828263.782541:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828263.782558:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828263/real 1787828263] req@ffff8ab3f075cb40 x1874671280739968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828263.782574:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280739968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828263.782585:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828263.782589:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828263.782593:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280739968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828263.782601:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280740096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828263.782621:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828263.782632:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828263/real 1787828263] req@ffff8ab3f075ed00 x1874671280740096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828263.782646:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280740096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828263.782656:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828263.782659:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828263.782664:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280740096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828263.782686:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280740224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828263.782732:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828263.782780:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828263/real 1787828263] req@ffff8ab3f075f480 x1874671280740224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828263.782806:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280740224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828263.782826:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828263.782833:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828263.782840:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280740224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828263.782854:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280740352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828263.782927:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828263.782946:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828263/real 1787828263] req@ffff8ab3f075da40 x1874671280740352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828263.782962:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280740352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828263.782976:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828263.782980:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828263.782984:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280740352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828264.806195:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828265.830097:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828266.854181:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828267.878117:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828268.902014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828268.902030:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828268.902037:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828268.902051:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2818 00000100:00080000:2.0:1787828268.902067:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828268.902078:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828268.902127:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828268.902143:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280740480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828268.902180:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828268.902185:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828268.902187:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828268.902191:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2818 00000100:00080000:2.0:1787828268.902196:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828268.902200:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828268.902215:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828268.902238:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280740608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828268.902239:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280740480:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828268.902261:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828268.902265:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828268.902267:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828268.902271:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2818 00000100:00080000:2.0:1787828268.902273:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828268.902278:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828268.902283:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828268.902296:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280740736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828268.902325:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828268.902329:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828268.902332:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828268.902336:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2818 00000100:00080000:2.0:1787828268.902354:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828268.902361:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828268.902369:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828268.902372:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828268.902408:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280740864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828268.902421:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828268.902425:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828268.902427:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828268.902432:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2818 00000100:00080000:2.0:1787828268.902436:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828268.902440:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828268.902448:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828268.902463:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280740992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828268.902470:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828268/real 1787828268] req@ffff8ab3f043b840 x1874671280740480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828268.902501:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280740480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828268.902514:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828268.902519:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828268.902525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280740480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828268.902576:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280740608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828268.902607:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828268.902624:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828268/real 1787828268] req@ffff8ab3f0439e00 x1874671280740608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828268.902640:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280740608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828268.902651:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828268.902654:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828268.902658:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280740608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828268.902665:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280740736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828268.902684:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828268.902696:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828268/real 1787828268] req@ffff8ab3f043b480 x1874671280740736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828268.902756:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280740736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828268.902773:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828268.902776:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828268.902780:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280740736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828268.902786:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280740864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828268.902809:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828268.902820:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828268/real 1787828268] req@ffff8ab3f043a940 x1874671280740864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828268.902831:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280740864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828268.902842:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828268.902844:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828268.902847:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280740864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828268.902853:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280740992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828268.902869:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828268.902878:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828268/real 1787828268] req@ffff8ab3f043a1c0 x1874671280740992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828268.902918:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280740992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828268.902930:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828268.902933:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828268.902937:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280740992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828269.926139:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828270.950180:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828271.974181:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828272.998131:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828274.022017:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828274.022030:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828274.022035:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828274.022049:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2823 00000100:00080000:2.0:1787828274.022063:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828274.022073:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828274.022126:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828274.022127:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280741120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828274.022172:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828274.022186:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828274.022190:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828274.022199:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2823 00000100:00080000:2.0:1787828274.022207:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828274.022216:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828274.022233:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828274.022242:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280741120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828274.022267:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280741248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828274.022294:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828274.022300:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828274.022303:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828274.022310:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2823 00000100:00080000:2.0:1787828274.022340:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828274.022347:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828274.022355:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787828274.022356:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828274.022380:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280741376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828274.022395:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828274.022399:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828274.022401:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828274.022407:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2823 00000100:00080000:2.0:1787828274.022413:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828274.022419:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828274.022432:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828274.022442:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828274/real 1787828274] req@ffff8ab3f0439a40 x1874671280741120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828274.022449:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280741504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828274.022511:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828274.022534:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828274.022537:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828274.022545:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2823 00000100:00080000:2.0:1787828274.022550:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828274.022557:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280741120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828274.022559:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828274.022576:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787828274.022602:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828274.022603:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280741632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828274.022627:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828274.022649:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280741120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828274.022700:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280741248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828274.022727:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828274.022742:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828274/real 1787828274] req@ffff8ab3f043ad00 x1874671280741248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828274.022756:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280741248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828274.022765:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828274.022769:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828274.022773:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280741248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828274.022780:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280741376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828274.022796:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828274.022806:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828274/real 1787828274] req@ffff8ab3f04392c0 x1874671280741376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828274.022826:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280741376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828274.022836:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828274.022840:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828274.022843:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280741376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828274.022851:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280741504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828274.022869:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828274.022880:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828274/real 1787828274] req@ffff8ab3f04383c0 x1874671280741504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828274.022921:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280741504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828274.022934:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828274.022937:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828274.022941:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280741504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828274.022948:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280741632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828274.022968:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828274.022978:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828274/real 1787828274] req@ffff8ab3f043bc00 x1874671280741632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828274.022990:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280741632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828274.023003:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828274.023006:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828274.023010:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280741632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828275.046203:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828276.070145:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828277.094021:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828278.118069:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828279.142000:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828279.142011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828279.142016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828279.142020:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828279.142028:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2828 00000100:00080000:0.0:1787828279.142037:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828279.142042:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828279.142073:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280741760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828279.142095:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828279.142097:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828279.142099:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828279.142101:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2828 00000100:00080000:0.0:1787828279.142104:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828279.142107:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828279.142114:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828279.142129:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280741888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828279.142129:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280741760:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828279.142147:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828279.142150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828279.142151:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828279.142154:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2828 00000100:00080000:0.0:1787828279.142157:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828279.142160:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828279.142163:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828279.142171:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280742016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828279.142178:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828279.142180:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828279.142181:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828279.142186:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828279.142187:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2828 00000100:00080000:0.0:1787828279.142192:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828279.142195:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828279.142200:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828279.142207:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280742144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828279.142214:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828279.142216:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828279.142217:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828279.142220:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2828 00000100:00080000:0.0:1787828279.142224:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828279.142227:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828279.142230:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828279.142231:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828279/real 1787828279] req@ffff8ab3f075e580 x1874671280741760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828279.142242:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280742272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828279.142250:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280741760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828279.142259:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828279.142262:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828279.142266:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280741760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828279.142284:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280741888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828279.142297:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828279.142305:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828279/real 1787828279] req@ffff8ab3f075da40 x1874671280741888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828279.142313:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280741888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828279.142318:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828279.142320:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828279.142322:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280741888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828279.142327:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280742016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828279.142336:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828279.142341:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828279/real 1787828279] req@ffff8ab3f075f480 x1874671280742016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828279.142348:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280742016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828279.142353:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828279.142355:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828279.142357:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280742016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828279.142360:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280742144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828279.142369:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828279.142375:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828279/real 1787828279] req@ffff8ab3f075ed00 x1874671280742144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828279.142381:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280742144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828279.142387:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828279.142389:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828279.142391:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280742144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828279.142394:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280742272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828279.142403:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828279.142408:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828279/real 1787828279] req@ffff8ab3f075cb40 x1874671280742272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828279.142415:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280742272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828279.142421:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828279.142423:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828279.142424:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280742272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828280.166006:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828281.190085:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828282.214043:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828283.238100:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828284.261994:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828284.262006:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828284.262010:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828284.262021:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2833 00000100:00080000:0.0:1787828284.262032:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828284.262041:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828284.262075:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828284.262082:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280742400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828284.262110:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828284.262113:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828284.262115:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828284.262118:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2833 00000100:00080000:0.0:1787828284.262121:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828284.262125:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828284.262134:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828284.262153:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280742528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828284.262153:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280742400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828284.262173:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828284.262179:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828284.262181:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828284.262186:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2833 00000100:00080000:0.0:1787828284.262190:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828284.262193:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828284.262198:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828284.262208:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280742656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828284.262217:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828284.262219:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828284.262221:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828284.262223:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2833 00000100:00080000:0.0:1787828284.262226:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828284.262229:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828284.262232:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828284.262241:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280742784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828284.262249:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828284.262252:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828284.262254:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828284.262257:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2833 00000100:00080000:0.0:1787828284.262259:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828284.262261:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828284.262266:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828284.262268:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280742528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828284.262278:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280742912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828284.262300:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280742656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828284.262318:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280742784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828284.262336:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280742912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828285.286037:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828286.310065:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828287.334150:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828287.334193:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787828287.334244:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828287.334262:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828287.334303:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828287.334310:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828287.334317:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828287.334325:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787828287.334362:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828284/real 1787828287] req@ffff8ab3f075e940 x1874671280742912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.334393:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280742912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.334410:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.334416:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828287.334422:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280742912:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828287.334431:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828284/real 1787828287] req@ffff8ab3f075f0c0 x1874671280742528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.334454:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280742528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.334470:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.334475:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828287.334479:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280742528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828287.334487:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828284/real 1787828287] req@ffff8ab3f075de00 x1874671280742656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.334502:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280742656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.334512:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.334516:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828287.334519:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280742656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828287.334527:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828284/real 1787828287] req@ffff8ab3f075f840 x1874671280742784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.334541:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280742784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.334552:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.334556:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828287.334559:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280742784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828287.334567:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828284/real 1787828287] req@ffff8ab3f075cf00 x1874671280742400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.334606:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280742400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.334616:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.334619:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828287.334623:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280742400:192.168.203.124@tcp:250:kworker.0 00010000:00010000:3.0:1787828288.358105:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828289.382039:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828289.382115:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828289.382120:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828289.382124:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828289.382132:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2839 00000100:00080000:0.0:1787828289.382139:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828289.382145:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828289.382175:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280743040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828289.382196:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828289.382199:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828289.382201:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828289.382203:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2839 00000100:00080000:0.0:1787828289.382206:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828289.382208:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828289.382219:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828289.382238:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280743168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828289.382251:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280743040:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828289.382258:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828289.382263:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828289.382265:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828289.382272:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2839 00000100:00080000:0.0:1787828289.382276:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828289.382281:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828289.382287:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828289.382308:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280743296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828289.382318:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828289.382327:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828289.382332:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828289.382335:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828289.382341:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2839 00000100:00080000:0.0:1787828289.382346:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828289.382351:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828289.382358:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828289.382377:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828289/real 1787828289] req@ffff8ab3f075c000 x1874671280743040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828289.382378:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280743424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828289.382415:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828289.382419:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828289.382423:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828289.382424:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280743040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828289.382432:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2839 00000100:00080000:0.0:1787828289.382440:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828289.382442:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828289.382447:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787828289.382452:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000008:00100000:0.0:1787828289.382457:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828289.382458:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280743040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828289.382477:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280743552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828289.382495:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280743168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828289.382523:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828289.382538:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828289/real 1787828289] req@ffff8ab3f075cf00 x1874671280743168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828289.382552:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280743168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828289.382715:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828289.382720:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828289.382724:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280743168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828289.382731:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280743296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828289.382755:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828289.382768:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828289/real 1787828289] req@ffff8ab3f075f840 x1874671280743296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828289.382781:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280743296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828289.382791:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828289.382794:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828289.382798:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280743296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828289.382804:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280743424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828289.382826:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828289.382838:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828289/real 1787828289] req@ffff8ab3f075de00 x1874671280743424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828289.382850:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280743424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828289.382861:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828289.382865:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828289.382869:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280743424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828289.382876:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280743552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828289.382918:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828289.382931:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828289/real 1787828289] req@ffff8ab3f075f0c0 x1874671280743552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828289.382945:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280743552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828289.382957:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828289.382961:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828289.382965:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280743552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828290.406075:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828291.430080:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828292.454080:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828293.478012:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828294.501976:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828294.501984:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828294.501988:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828294.501997:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2844 00010000:00010000:2.0:1787828294.502006:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828294.502007:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828294.502014:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828294.502046:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280743680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828294.502065:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828294.502068:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828294.502069:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828294.502072:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2844 00000100:00080000:0.0:1787828294.502074:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828294.502077:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828294.502085:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828294.502093:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280743808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828294.502098:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280743680:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828294.502113:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828294.502115:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828294.502117:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828294.502121:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2844 00000100:00080000:0.0:1787828294.502123:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828294.502125:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828294.502129:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828294.502137:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280743936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828294.502144:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828294.502146:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828294.502148:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828294.502150:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2844 00000100:00080000:0.0:1787828294.502152:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828294.502155:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828294.502160:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828294.502161:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828294.502170:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280744064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828294.502177:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828294.502179:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828294.502180:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828294.502183:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2844 00000100:00080000:0.0:1787828294.502185:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828294.502189:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828294.502192:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828294.502201:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828294/real 1787828294] req@ffff8ab3f075e940 x1874671280743680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828294.502205:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280744192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828294.502230:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280743680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828294.502240:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828294.502244:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828294.502246:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280743680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828294.502263:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280743808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828294.502276:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828294.502285:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828294/real 1787828294] req@ffff8ab3f075cb40 x1874671280743808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828294.502293:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280743808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828294.502298:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828294.502301:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828294.502303:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280743808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828294.502306:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280743936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828294.502326:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828294.502333:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828294/real 1787828294] req@ffff8ab3f075ed00 x1874671280743936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828294.502340:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280743936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828294.502346:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828294.502348:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828294.502350:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280743936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828294.502353:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280744064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828294.502363:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828294.502369:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828294/real 1787828294] req@ffff8ab3f075f480 x1874671280744064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828294.502375:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280744064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828294.502382:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828294.502384:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828294.502386:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280744064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828294.502389:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280744192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828294.502398:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828294.502403:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828294/real 1787828294] req@ffff8ab3f075da40 x1874671280744192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828294.502409:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280744192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828294.502415:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828294.502417:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828294.502419:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280744192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828295.526013:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828296.550016:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828297.574035:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828298.598019:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828299.621991:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828299.622001:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828299.622006:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828299.622017:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2849 00000100:00080000:0.0:1787828299.622028:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828299.622036:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828299.622041:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828299.622074:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280744320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828299.622099:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828299.622103:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828299.622105:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828299.622109:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2849 00000100:00080000:0.0:1787828299.622114:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828299.622119:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828299.622129:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828299.622129:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280744320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828299.622159:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280744448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828299.622172:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828299.622179:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828299.622181:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828299.622188:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2849 00000800:00000100:1.0:1787828299.622189:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828299.622196:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828299.622201:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828299.622208:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828299.622221:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280744576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828299.622231:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828299.622234:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828299.622236:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828299.622240:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828299/real 1787828299] req@ffff8ab3f075e580 x1874671280744320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828299.622244:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2849 00000100:00080000:0.0:1787828299.622251:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828299.622259:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828299.622263:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280744320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828299.622266:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787828299.622277:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828299.622281:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828299.622286:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280744704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828299.622288:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280744320:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828299.622302:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828299.622306:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828299.622308:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828299.622313:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280744448:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787828299.622315:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2849 00000100:00080000:0.0:1787828299.622321:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828299.622327:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828299.622333:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787828299.622333:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828299.622346:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828299/real 1787828299] req@ffff8ab3f075da40 x1874671280744448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828299.622352:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280744832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828299.622363:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280744448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828299.622372:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828299.622376:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828299.622378:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280744448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828299.622383:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280744576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828299.622395:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828299.622401:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828299/real 1787828299] req@ffff8ab3f075f480 x1874671280744576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828299.622408:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280744576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828299.622413:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828299.622414:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828299.622416:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280744576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828299.622420:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280744704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828299.622430:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828299.622436:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828299/real 1787828299] req@ffff8ab3f075ed00 x1874671280744704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828299.622443:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280744704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828299.622449:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828299.622450:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828299.622452:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280744704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828299.622462:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280744832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828299.622472:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828299.622478:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828299/real 1787828299] req@ffff8ab3f075cb40 x1874671280744832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828299.622498:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280744832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828299.622505:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828299.622507:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828299.622509:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280744832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828300.646069:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828301.670020:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828302.694010:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828303.718010:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828304.741992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828304.742003:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828304.742007:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828304.742017:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2854 00000100:00080000:2.0:1787828304.742027:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828304.742035:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828304.742059:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828304.742072:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280744960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828304.742099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828304.742103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828304.742105:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828304.742109:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2854 00000100:00080000:2.0:1787828304.742113:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828304.742118:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828304.742128:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828304.742133:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280744960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828304.742146:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280745088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828304.742158:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828304.742162:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828304.742164:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828304.742169:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2854 00000100:00080000:2.0:1787828304.742180:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828304.742186:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828304.742189:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787828304.742194:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828304.742209:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280745216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828304.742220:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828304.742224:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828304.742226:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828304.742232:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828304/real 1787828304] req@ffff8ab3f0439680 x1874671280744960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828304.742233:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2854 00000100:00080000:2.0:1787828304.742241:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828304.742249:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828304.742256:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280744960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828304.742256:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787828304.742270:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828304.742274:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280745344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828304.742276:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828304.742284:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280744960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828304.742297:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828304.742300:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828304.742302:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828304.742308:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2854 00000100:00100000:1.0:1787828304.742312:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280745088:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787828304.742314:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828304.742322:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828304.742327:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787828304.742334:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828304.742343:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280745472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828304.742348:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828304/real 1787828304] req@ffff8ab3f043a580 x1874671280745088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828304.742374:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280745088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828304.742389:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828304.742392:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828304.742395:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280745088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828304.742400:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280745216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828304.742413:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828304.742419:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828304/real 1787828304] req@ffff8ab3f0438b40 x1874671280745216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828304.742426:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280745216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828304.742431:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828304.742433:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828304.742435:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280745216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828304.742439:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280745344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828304.742463:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828304.742472:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828304/real 1787828304] req@ffff8ab3f0438780 x1874671280745344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828304.742479:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280745344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828304.742495:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828304.742503:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828304.742507:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280745344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828304.742522:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280745472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828304.742543:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828304.742554:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828304/real 1787828304] req@ffff8ab3f043b0c0 x1874671280745472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828304.742562:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280745472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828304.742569:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828304.742571:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828304.742573:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280745472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828305.766078:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828306.790099:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828307.814066:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828308.838053:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828309.862017:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828309.862044:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828309.862072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828309.862079:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828309.862094:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2859 00000100:00080000:0.0:1787828309.862117:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828309.862128:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828309.862190:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280745600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828309.862264:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828309.862269:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828309.862272:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828309.862277:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2859 00000100:00080000:0.0:1787828309.862281:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828309.862286:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828309.862299:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828309.862315:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280745728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828309.862328:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828309.862332:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828309.862334:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828309.862338:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2859 00000100:00080000:0.0:1787828309.862343:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828309.862349:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828309.862355:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828309.862366:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280745856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828309.862378:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828309.862383:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828309.862386:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828309.862393:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280745600:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828309.862395:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2859 00000100:00080000:0.0:1787828309.862405:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828309.862412:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828309.862437:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828309.862451:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280745984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828309.862462:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828309.862467:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828309.862470:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828309.862474:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2859 00000100:00080000:0.0:1787828309.862478:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828309.862483:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828309.862489:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828309.862501:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280746112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828309.862541:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828309.862611:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828309/real 1787828309] req@ffff8ab3f075e940 x1874671280745600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828309.862636:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280745600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828309.862650:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828309.862656:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828309.862661:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280745600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828309.862670:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280745728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828309.862693:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828309.862706:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828309/real 1787828309] req@ffff8ab3f075f0c0 x1874671280745728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828309.862720:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280745728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828309.862730:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828309.862733:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828309.862737:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280745728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828309.862768:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280745856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828309.862787:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828309.862797:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828309/real 1787828309] req@ffff8ab3f075de00 x1874671280745856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828309.862811:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280745856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828309.862828:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828309.862832:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828309.862835:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280745856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828309.862842:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280745984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828309.862877:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828309.862889:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828309/real 1787828309] req@ffff8ab3f075f840 x1874671280745984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828309.862945:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280745984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828309.862960:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828309.862963:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828309.862967:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280745984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828309.862975:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280746112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828309.863001:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828309.863013:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828309/real 1787828309] req@ffff8ab3f075cf00 x1874671280746112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828309.863027:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280746112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828309.863039:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828309.863042:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828309.863047:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280746112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828310.886132:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828311.910026:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828312.934100:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828313.958036:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828314.982020:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828314.982031:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828314.982037:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828314.982049:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2864 00000100:00080000:2.0:1787828314.982062:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828314.982072:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828314.982093:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828314.982118:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280746240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828314.982163:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828314.982177:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828314.982180:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828314.982185:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2864 00000100:00080000:2.0:1787828314.982189:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828314.982193:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280746240:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828314.982196:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828314.982209:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828314.982227:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280746368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828314.982267:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828314.982272:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828314.982276:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828314.982283:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828314.982288:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2864 00000100:00080000:2.0:1787828314.982295:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828314.982301:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828314.982317:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828314.982346:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280746496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828314.982353:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828314.982357:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828314/real 1787828314] req@ffff8ab3f0438f00 x1874671280746240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828314.982358:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828314.982362:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828314.982370:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2864 00000100:00080000:2.0:1787828314.982375:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828314.982391:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828314.982393:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280746240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828314.982397:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787828314.982418:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828314.982422:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828314.982425:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280746624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828314.982428:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280746240:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828314.982458:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828314.982461:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828314.982463:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828314.982468:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2864 00000100:00080000:2.0:1787828314.982483:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828314.982484:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280746368:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787828314.982492:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828314.982498:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828314.982511:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280746752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828314.982518:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828314.982538:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828314/real 1787828314] req@ffff8ab3f043b0c0 x1874671280746368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828314.982552:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280746368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828314.982560:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828314.982562:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828314.982565:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280746368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828314.982571:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280746496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828314.982599:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828314.982617:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828314/real 1787828314] req@ffff8ab3f0438780 x1874671280746496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828314.982624:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280746496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828314.982630:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828314.982632:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828314.982634:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280746496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828314.982637:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280746624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828314.982647:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828314.982653:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828314/real 1787828314] req@ffff8ab3f0438b40 x1874671280746624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828314.982660:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280746624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828314.982666:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828314.982667:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828314.982669:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280746624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828314.982678:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280746752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828314.982687:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828314.982693:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828314/real 1787828314] req@ffff8ab3f043a580 x1874671280746752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828314.982700:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280746752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828314.982706:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828314.982708:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828314.982710:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280746752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828316.006057:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828317.030188:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828318.054085:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828319.078196:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828320.102045:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828320.102053:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828320.102056:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828320.102064:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2869 00000100:00080000:0.0:1787828320.102073:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828320.102078:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828320.102111:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828320.102113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280746880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828320.102135:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828320.102137:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828320.102139:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828320.102141:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2869 00000100:00080000:0.0:1787828320.102144:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828320.102147:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828320.102155:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828320.102165:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280747008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828320.102170:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280746880:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828320.102178:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828320.102181:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828320.102182:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828320.102186:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2869 00000100:00080000:0.0:1787828320.102188:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828320.102190:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828320.102195:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828320.102205:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280747136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828320.102212:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828320.102214:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828320.102215:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828320.102217:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2869 00000100:00080000:0.0:1787828320.102220:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828320.102223:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828320.102226:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828320.102230:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828320.102235:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280747264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828320.102243:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828320.102245:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828320.102247:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828320.102250:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2869 00000100:00080000:0.0:1787828320.102252:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828320.102254:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828320.102257:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828320.102266:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280747392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828320.102275:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828320/real 1787828320] req@ffff8ab3f075c000 x1874671280746880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828320.102289:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280746880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828320.102298:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828320.102301:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828320.102304:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280746880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828320.102321:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280747008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828320.102335:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828320.102360:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828320/real 1787828320] req@ffff8ab3f075cf00 x1874671280747008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828320.102369:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280747008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828320.102375:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828320.102377:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828320.102379:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280747008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828320.102383:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280747136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828320.102395:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828320.102402:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828320/real 1787828320] req@ffff8ab3f075f840 x1874671280747136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828320.102409:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280747136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828320.102414:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828320.102416:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828320.102417:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280747136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828320.102421:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280747264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828320.102431:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828320.102436:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828320/real 1787828320] req@ffff8ab3f075de00 x1874671280747264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828320.102443:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280747264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828320.102449:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828320.102451:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828320.102453:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280747264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828320.102456:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280747392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828320.102465:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828320.102471:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828320/real 1787828320] req@ffff8ab3f075f0c0 x1874671280747392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828320.102478:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280747392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828320.102484:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828320.102485:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828320.102487:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280747392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828321.126146:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828322.150184:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828323.174124:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828324.198190:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828325.221987:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828325.221999:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828325.222005:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828325.222015:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2874 00000100:00080000:0.0:1787828325.222025:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828325.222075:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828325.222126:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828325.222141:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280747520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828325.222189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828325.222194:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828325.222197:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828325.222203:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2874 00000100:00080000:0.0:1787828325.222208:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828325.222232:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828325.222258:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280747520:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787828325.222261:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828325.222297:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280747648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828325.222361:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828325.222365:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828325.222366:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828325.222369:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2874 00000100:00080000:0.0:1787828325.222374:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828325.222378:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828325.222383:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828325.222385:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828325.222417:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280747776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828325.222441:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828325.222446:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828325.222449:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828325.222456:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2874 00000100:00080000:0.0:1787828325.222461:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828325.222483:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828325/real 1787828325] req@ffff8ab3f075e940 x1874671280747520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828325.222487:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828325.222496:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828325.222508:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280747520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828325.222510:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280747904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828325.222534:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828325.222536:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828325.222541:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828325.222542:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787828325.222546:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828325.222550:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280747520:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828325.222563:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2874 00000100:00080000:0.0:1787828325.222570:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828325.222593:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828325.222599:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828325.222614:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280748032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828325.222620:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280747648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828325.222660:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828325.222684:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828325/real 1787828325] req@ffff8ab3f075cb40 x1874671280747648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828325.222722:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280747648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828325.222735:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828325.222739:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828325.222743:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280747648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828325.222750:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280747776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828325.222771:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828325.222784:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828325/real 1787828325] req@ffff8ab3f075ed00 x1874671280747776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828325.222825:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280747776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828325.222837:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828325.222841:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828325.222845:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280747776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828325.222854:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280747904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828325.222882:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828325.222896:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828325/real 1787828325] req@ffff8ab3f075f480 x1874671280747904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828325.222910:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280747904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828325.222921:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828325.222925:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828325.222930:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280747904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828325.222951:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280748032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828325.222970:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828325.223017:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828325/real 1787828325] req@ffff8ab3f075da40 x1874671280748032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828325.223026:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280748032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828325.223034:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828325.223036:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828325.223039:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280748032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828326.246093:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828327.270015:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828328.294002:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828329.318040:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828330.342043:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828330.342051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828330.342055:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1787828330.342059:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828330.342064:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2879 00000100:00080000:1.0:1787828330.342073:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828330.342079:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828330.342109:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280748160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828330.342130:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828330.342143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828330.342144:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828330.342147:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2879 00000100:00080000:1.0:1787828330.342149:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828330.342152:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828330.342160:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828330.342169:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280748288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828330.342177:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280748160:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787828330.342179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828330.342185:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828330.342188:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828330.342192:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2879 00000100:00080000:1.0:1787828330.342194:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828330.342197:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828330.342208:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828330.342216:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280748416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828330.342222:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828330.342225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828330.342226:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828330.342229:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2879 00000100:00080000:1.0:1787828330.342244:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828330.342247:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828330.342251:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787828330.342256:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828330.342262:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280748544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828330.342292:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828330.342295:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828330.342297:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828330.342300:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2879 00000100:00080000:1.0:1787828330.342303:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828330.342305:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828330.342308:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828330.342318:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280748672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828330.342341:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828330/real 1787828330] req@ffff8ab3d22fd680 x1874671280748160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828330.342370:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280748160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828330.342381:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828330.342385:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828330.342389:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280748160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828330.342410:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280748288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828330.342433:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828330.342448:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828330/real 1787828330] req@ffff8ab3d22fcb40 x1874671280748288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828330.342462:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280748288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828330.342476:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828330.342480:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828330.342483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280748288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828330.342489:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280748416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828330.342509:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828330.342520:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828330/real 1787828330] req@ffff8ab3d22fda40 x1874671280748416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828330.342532:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280748416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828330.342540:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828330.342543:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828330.342546:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280748416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828330.342551:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280748544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828330.342569:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828330.342581:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828330/real 1787828330] req@ffff8ab3d22fc000 x1874671280748544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828330.342595:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280748544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828330.342604:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828330.342608:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828330.342623:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280748544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828330.342629:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280748672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828330.342646:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828330.342657:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828330/real 1787828330] req@ffff8ab3d22ff480 x1874671280748672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828330.342669:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280748672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828330.342678:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828330.342681:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828330.342684:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280748672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828331.366106:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828332.390153:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828333.414177:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828334.438075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828335.462012:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828335.462023:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828335.462028:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828335.462040:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2885 00000100:00080000:0.0:1787828335.462054:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828335.462062:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828335.462070:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828335.462114:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280748800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828335.462141:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828335.462144:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828335.462146:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828335.462149:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2885 00000100:00080000:0.0:1787828335.462152:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828335.462154:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828335.462165:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828335.462177:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280748928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828335.462184:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828335.462186:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828335.462188:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828335.462191:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2885 00000100:00080000:0.0:1787828335.462193:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828335.462197:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828335.462200:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828335.462208:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280749056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828335.462216:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828335.462220:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828335.462223:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828335.462224:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280748800:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828335.462228:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2885 00000100:00080000:0.0:1787828335.462233:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828335.462251:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828335.462257:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828335.462271:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280749184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828335.462304:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828335.462311:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828335.462314:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828335.462320:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2885 00000100:00080000:0.0:1787828335.462342:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828335.462348:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828335.462355:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:2.0:1787828335.462363:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828335.462377:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280749312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828335.462444:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828335/real 1787828335] req@ffff8ab3f075e580 x1874671280748800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828335.462468:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280748800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828335.462497:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828335.462501:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828335.462504:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280748800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828335.462511:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280748928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828335.462529:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828335.462539:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828335/real 1787828335] req@ffff8ab3f075da40 x1874671280748928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828335.462547:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280748928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828335.462553:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828335.462555:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828335.462557:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280748928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828335.462585:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280749056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828335.462605:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828335.462618:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828335/real 1787828335] req@ffff8ab3f075f480 x1874671280749056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828335.462632:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280749056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828335.462642:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828335.462646:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828335.462649:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280749056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828335.462654:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280749184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828335.462672:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828335.462689:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828335/real 1787828335] req@ffff8ab3f075ed00 x1874671280749184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828335.462703:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280749184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828335.462712:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828335.462714:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828335.462716:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280749184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828335.462720:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280749312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828335.462733:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828335.462739:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828335/real 1787828335] req@ffff8ab3f075cb40 x1874671280749312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828335.462747:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280749312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828335.462754:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828335.462756:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828335.462758:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280749312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828336.486102:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828337.510184:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828338.534161:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828339.558161:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828340.582047:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828340.582060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828340.582066:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828340.582081:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2890 00000100:00080000:0.0:1787828340.582095:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828340.582105:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828340.582155:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280749440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828340.582173:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828340.582219:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828340.582227:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828340.582231:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828340.582239:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2890 00000100:00080000:0.0:1787828340.582245:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828340.582252:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828340.582265:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828340.582289:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280749568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828340.582295:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280749440:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828340.582323:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828340.582329:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828340.582331:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828340.582338:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2890 00000100:00080000:0.0:1787828340.582342:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828340.582347:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828340.582353:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828340.582368:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280749696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828340.582380:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828340.582385:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828340.582387:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828340.582392:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2890 00000100:00080000:0.0:1787828340.582414:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828340.582421:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828340.582429:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828340.582431:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828340.582455:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280749824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828340.582469:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828340.582474:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828340.582477:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828340.582482:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2890 00000100:00080000:0.0:1787828340.582488:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828340.582493:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828340.582502:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828340.582525:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280749952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828340.582539:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828340/real 1787828340] req@ffff8ab3f075e940 x1874671280749440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828340.582573:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280749440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828340.582590:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828340.582596:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828340.582602:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280749440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828340.582641:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280749568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828340.582670:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828340.582685:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828340/real 1787828340] req@ffff8ab3f075f0c0 x1874671280749568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828340.582701:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280749568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828340.582729:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828340.582735:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828340.582739:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280749568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828340.582748:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280749696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828340.582773:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828340.582785:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828340/real 1787828340] req@ffff8ab3f075de00 x1874671280749696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828340.582800:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280749696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828340.582810:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828340.582815:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828340.582818:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280749696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828340.582826:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280749824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828340.582846:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828340.582857:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828340/real 1787828340] req@ffff8ab3f075f840 x1874671280749824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828340.582871:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280749824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828340.582922:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828340.582927:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828340.582933:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280749824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828340.582941:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280749952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828340.582965:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828340.582979:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828340/real 1787828340] req@ffff8ab3f075cf00 x1874671280749952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828340.582993:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280749952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828340.583006:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828340.583010:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828340.583014:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280749952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828341.606234:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828342.630199:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828343.654090:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828344.678133:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828345.702030:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828345.702040:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828345.702045:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828345.702056:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2895 00000100:00080000:0.0:1787828345.702067:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828345.702077:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828345.702118:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280750080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828345.702137:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828345.702150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828345.702159:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828345.702163:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828345.702191:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2895 00000100:00080000:0.0:1787828345.702198:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828345.702203:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828345.702214:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280750080:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787828345.702215:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828345.702248:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280750208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828345.702268:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828345.702274:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828345.702278:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828345.702285:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2895 00000800:00000100:1.0:1787828345.702292:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828345.702295:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828345.702307:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828345.702316:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828345.702334:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280750336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828345.702346:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828345.702348:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828345/real 1787828345] req@ffff8ab3f075c000 x1874671280750080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828345.702351:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828345.702357:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828345.702371:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2895 00000100:00080000:0.0:1787828345.702378:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828345.702406:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828345.702408:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280750080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828345.702418:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828345.702442:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280750464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828345.702444:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828345.702463:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787828345.702467:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828345.702493:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828345.702495:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280750080:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828345.702498:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828345.702511:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2895 00000100:00080000:0.0:1787828345.702517:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828345.702522:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828345.702528:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828345.702558:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280750592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828345.702561:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280750208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828345.702604:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828345.702636:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828345/real 1787828345] req@ffff8ab3f075cf00 x1874671280750208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828345.702647:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280750208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828345.702654:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828345.702657:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828345.702660:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280750208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828345.702664:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280750336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828345.702678:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828345.702695:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828345/real 1787828345] req@ffff8ab3f075f840 x1874671280750336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828345.702705:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280750336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828345.702712:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828345.702714:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828345.702717:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280750336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828345.702722:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280750464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828345.702735:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828345.702743:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828345/real 1787828345] req@ffff8ab3f075de00 x1874671280750464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828345.702751:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280750464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828345.702759:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828345.702762:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828345.702765:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280750464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828345.702778:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280750592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828345.702789:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828345.702796:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828345/real 1787828345] req@ffff8ab3f075f0c0 x1874671280750592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828345.702804:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280750592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828345.702812:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828345.702814:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828345.702817:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280750592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828346.726082:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828347.750320:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828348.774209:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828349.798163:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828350.822115:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828350.822153:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828350.822161:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828350.822179:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2900 00000100:00080000:0.0:1787828350.822196:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828350.822211:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828350.822286:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828350.822290:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280750720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828350.822343:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828350.822347:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828350.822350:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828350.822355:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2900 00000100:00080000:0.0:1787828350.822372:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828350.822384:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828350.822401:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828350.822420:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280750720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828350.822424:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280750848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828350.822458:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828350.822462:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828350.822465:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828350.822481:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2900 00000100:00080000:0.0:1787828350.822485:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828350.822490:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828350.822500:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828350.822524:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280750976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828350.822551:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828350.822559:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828350.822563:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828350.822571:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2900 00000100:00080000:0.0:1787828350.822581:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828350.822592:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828350.822606:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828350.822632:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280751104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828350.822639:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280750848:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787828350.822704:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828350.822710:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828350.822714:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828350.822723:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2900 00000100:00080000:0.0:1787828350.822735:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828350.822746:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280750976:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787828350.822747:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828350.822772:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828350.822810:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280751232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828350.822846:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280751104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828350.822944:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280751232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828351.846163:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828352.870081:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828353.894338:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787828353.894414:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787828353.894504:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828353.894533:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787828353.894608:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787828353.894629:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787828353.894665:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787828353.894688:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828350/real 1787828353] req@ffff8ab3f075cb40 x1874671280750848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787828353.894689:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787828353.894748:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280750848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828353.894767:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828353.894773:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828353.894779:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280750848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828353.894789:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828350/real 1787828353] req@ffff8ab3f075ed00 x1874671280750976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828353.894806:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280750976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828353.894817:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828353.894822:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828353.894826:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280750976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828353.894836:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828350/real 1787828353] req@ffff8ab3f075e940 x1874671280750720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828353.894854:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280750720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828353.894867:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828353.894872:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828353.894877:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280750720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828353.894945:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828350/real 1787828353] req@ffff8ab3f075f480 x1874671280751104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828353.894972:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280751104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828353.894990:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828353.894997:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828353.895002:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280751104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828353.895014:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828350/real 1787828353] req@ffff8ab3f075da40 x1874671280751232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828353.895033:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280751232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828353.895048:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828353.895055:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828353.895060:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280751232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828354.918133:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828355.941990:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828355.942001:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828355.942005:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828355.942016:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2905 00000100:00080000:0.0:1787828355.942026:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828355.942034:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828355.942034:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828355.942070:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280751360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828355.942103:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828355.942106:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828355.942108:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828355.942111:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2905 00000100:00080000:0.0:1787828355.942114:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828355.942117:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828355.942126:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828355.942138:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280751488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828355.942146:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280751360:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828355.942148:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828355.942165:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828355.942166:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828355.942170:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2905 00000100:00080000:0.0:1787828355.942172:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828355.942174:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828355.942178:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828355.942186:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280751616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828355.942192:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828355.942195:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828355.942197:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828355.942199:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2905 00000100:00080000:0.0:1787828355.942208:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828355.942211:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828355.942215:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828355.942216:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828355.942226:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280751744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828355.942233:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828355.942235:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828355.942237:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828355.942240:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2905 00000100:00080000:0.0:1787828355.942242:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828355.942245:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828355.942249:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828355.942257:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828355/real 1787828355] req@ffff8ab3f075e580 x1874671280751360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828355.942259:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280751872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828355.942295:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280751360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828355.942304:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828355.942308:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828355.942313:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280751360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828355.942344:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280751488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828355.942367:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828355.942381:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828355/real 1787828355] req@ffff8ab3f075da40 x1874671280751488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828355.942403:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280751488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828355.942417:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828355.942421:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828355.942425:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280751488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828355.942432:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280751616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828355.942453:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828355.942462:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828355/real 1787828355] req@ffff8ab3f075f480 x1874671280751616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828355.942475:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280751616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828355.942490:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828355.942494:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828355.942497:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280751616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828355.942504:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280751744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828355.942525:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828355.942538:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828355/real 1787828355] req@ffff8ab3f075e940 x1874671280751744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828355.942561:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280751744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828355.942571:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828355.942574:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828355.942578:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280751744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828355.942584:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280751872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828355.942603:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828355.942614:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828355/real 1787828355] req@ffff8ab3f075ed00 x1874671280751872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828355.942625:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280751872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828355.942635:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828355.942638:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828355.942642:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280751872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828356.966200:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828357.990096:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828359.014198:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828360.038023:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828361.062033:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828361.062046:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828361.062052:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828361.062084:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2910 00000100:00080000:0.0:1787828361.062119:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828361.062129:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828361.062177:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828361.062191:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280752000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828361.062225:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828361.062229:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828361.062231:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828361.062234:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2910 00000100:00080000:0.0:1787828361.062237:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828361.062240:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828361.062254:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828361.062270:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280752128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828361.062287:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828361.062289:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828361.062291:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828361.062299:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2910 00000100:00080000:0.0:1787828361.062306:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828361.062307:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280752000:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828361.062317:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828361.062326:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828361.062339:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280752256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828361.062349:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828361.062353:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828361.062355:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828361.062359:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2910 00000100:00080000:0.0:1787828361.062361:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828361.062366:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828361.062371:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828361.062383:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280752384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828361.062390:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828361.062394:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828361.062395:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828361.062400:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2910 00000100:00080000:0.0:1787828361.062404:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828361.062412:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828361.062419:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828361.062425:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828361.062442:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280752512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828361.062500:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828361/real 1787828361] req@ffff8ab3f075cb40 x1874671280752000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828361.062539:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280752000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828361.062563:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828361.062571:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828361.062577:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280752000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828361.062610:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280752128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828361.062646:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828361.062664:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828361/real 1787828361] req@ffff8ab3f075f0c0 x1874671280752128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828361.062699:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280752128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828361.062711:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828361.062715:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828361.062720:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280752128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828361.062728:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280752256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828361.062751:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828361.062764:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828361/real 1787828361] req@ffff8ab3f075de00 x1874671280752256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828361.062778:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280752256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828361.062788:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828361.062792:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828361.062797:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280752256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828361.062805:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280752384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828361.062824:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828361.062835:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828361/real 1787828361] req@ffff8ab3f075f840 x1874671280752384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828361.062848:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280752384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828361.062865:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828361.062872:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828361.062878:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280752384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828361.062936:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280752512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828361.062978:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828361.063007:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828361/real 1787828361] req@ffff8ab3f075cf00 x1874671280752512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828361.063022:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280752512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828361.063034:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828361.063039:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828361.063043:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280752512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828362.086118:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828363.110126:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828364.134094:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828365.158198:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828366.182054:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828366.182067:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828366.182074:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828366.182089:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2915 00010000:00010000:3.0:1787828366.182091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828366.182106:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828366.182117:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828366.182166:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280752640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828366.182201:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828366.182205:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828366.182207:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828366.182210:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2915 00000100:00080000:1.0:1787828366.182213:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828366.182217:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828366.182229:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828366.182241:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280752768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828366.182251:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828366.182255:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828366.182256:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828366.182261:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2915 00000100:00080000:1.0:1787828366.182264:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828366.182269:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828366.182274:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828366.182288:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280752896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828366.182291:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280752640:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787828366.182327:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828366.182330:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828366.182338:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828366.182344:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2915 00000100:00080000:1.0:1787828366.182347:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828366.182352:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828366.182359:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828366.182375:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280753024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828366.182385:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828366.182389:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828366.182390:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828366.182394:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2915 00000100:00080000:1.0:1787828366.182401:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828366.182402:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:1.0:1787828366.182414:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828366.182424:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828366.182441:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280753152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828366.182526:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828366/real 1787828366] req@ffff8ab3d22ff480 x1874671280752640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828366.182567:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280752640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828366.182587:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828366.182593:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828366.182599:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280752640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828366.182611:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280752768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828366.182647:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828366.182666:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828366/real 1787828366] req@ffff8ab3d22fc000 x1874671280752768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828366.182688:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280752768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828366.182706:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828366.182713:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828366.182719:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280752768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828366.182760:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280752896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828366.182800:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828366.182823:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828366/real 1787828366] req@ffff8ab3d22fda40 x1874671280752896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828366.182848:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280752896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828366.182865:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828366.182872:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828366.182880:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280752896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828366.182937:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280753024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828366.182969:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828366.182986:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828366/real 1787828366] req@ffff8ab3d22fcb40 x1874671280753024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828366.183001:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280753024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828366.183052:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828366.183058:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828366.183063:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280753024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828366.183080:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280753152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828366.183108:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828366.183122:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828366/real 1787828366] req@ffff8ab3d22fd680 x1874671280753152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828366.183137:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280753152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828366.183149:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828366.183153:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828366.183157:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280753152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828367.206210:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828368.230132:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828369.254237:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828370.278055:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828371.302111:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828371.302123:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828371.302129:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828371.302143:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2920 00010000:00010000:2.0:1787828371.302152:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828371.302157:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828371.302168:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828371.302215:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280753280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828371.302254:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828371.302259:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828371.302262:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828371.302267:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2920 00000100:00080000:0.0:1787828371.302272:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828371.302277:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828371.302291:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828371.302301:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280753280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828371.302312:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280753408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828371.302329:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828371.302337:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828371.302351:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828371.302358:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2920 00000100:00080000:0.0:1787828371.302372:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828371.302378:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787828371.302387:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828371.302393:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828371.302423:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280753536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828371.302443:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828371.302452:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828371/real 1787828371] req@ffff8ab3f075c000 x1874671280753280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828371.302454:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828371.302462:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828371.302497:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2920 00000100:00080000:0.0:1787828371.302510:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828371.302522:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828371.302530:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280753280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828371.302535:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828371.302561:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280753664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828371.302592:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828371.302597:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828371.302605:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787828371.302611:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828371.302619:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280753280:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828371.302621:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828371.302640:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2920 00000100:00080000:0.0:1787828371.302650:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828371.302658:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828371.302670:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828371.302672:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280753408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828371.302700:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280753792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828371.302720:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828371.302759:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828371/real 1787828371] req@ffff8ab3f075cf00 x1874671280753408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828371.302776:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280753408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828371.302787:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828371.302791:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828371.302795:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280753408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828371.302802:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280753536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828371.302823:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828371.302834:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828371/real 1787828371] req@ffff8ab3f075f840 x1874671280753536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828371.302850:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280753536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828371.302866:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828371.302872:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828371.302879:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280753536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828371.302942:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280753664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828371.302974:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828371.303017:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828371/real 1787828371] req@ffff8ab3f075de00 x1874671280753664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828371.303034:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280753664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828371.303045:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828371.303049:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828371.303053:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280753664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828371.303074:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280753792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828371.303099:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828371.303111:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828371/real 1787828371] req@ffff8ab3f075f0c0 x1874671280753792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828371.303124:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280753792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828371.303135:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828371.303139:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828371.303142:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280753792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828372.326015:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828373.350188:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828374.374194:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828375.398094:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828376.422112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828376.422129:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828376.422137:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828376.422153:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2926 00000100:00080000:2.0:1787828376.422169:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828376.422181:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828376.422236:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280753920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828376.422264:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828376.422290:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828376.422299:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828376.422316:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828376.422326:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2926 00000100:00080000:2.0:1787828376.422335:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828376.422345:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828376.422362:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280753920:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787828376.422363:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828376.422404:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280754048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828376.422429:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828376.422436:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828376.422440:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828376.422446:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2926 00000100:00080000:2.0:1787828376.422453:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828376.422455:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828376.422465:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828376.422476:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828376.422503:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280754176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828376.422523:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828376/real 1787828376] req@ffff8ab3f0439680 x1874671280753920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828376.422528:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828376.422547:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828376.422555:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828376.422571:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2926 00000100:00080000:2.0:1787828376.422583:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828376.422592:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280753920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828376.422597:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787828376.422613:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828376.422622:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000008:00100000:2.0:1787828376.422630:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828376.422631:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280753920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828376.422661:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280754304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828376.422674:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280754048:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787828376.422684:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828376.422691:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828376.422694:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828376.422702:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2926 00000100:00080000:2.0:1787828376.422719:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828376.422727:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828376.422732:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828376.422742:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828376.422768:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828376/real 1787828376] req@ffff8ab3f043bc00 x1874671280754048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828376.422771:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280754432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828376.422828:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280754048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828376.422841:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828376.422847:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828376.422851:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280754048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828376.422860:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280754176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828376.422924:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828376.422941:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828376/real 1787828376] req@ffff8ab3f04383c0 x1874671280754176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828376.422989:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280754176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828376.423009:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828376.423014:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828376.423018:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280754176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828376.423039:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280754304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828376.423069:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828376.423083:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828376/real 1787828376] req@ffff8ab3f04392c0 x1874671280754304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828376.423106:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280754304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828376.423125:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828376.423132:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828376.423138:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280754304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828376.423152:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280754432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828376.423191:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828376.423212:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828376/real 1787828376] req@ffff8ab3f043ad00 x1874671280754432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828376.423228:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280754432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828376.423240:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828376.423244:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828376.423248:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280754432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828377.446138:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828378.470139:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828379.494029:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828380.518106:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828381.542039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828381.542051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828381.542058:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828381.542073:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2931 00000100:00080000:2.0:1787828381.542085:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828381.542096:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828381.542147:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828381.542148:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280754560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828381.542199:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828381.542204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828381.542207:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828381.542212:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2931 00000100:00080000:2.0:1787828381.542217:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828381.542223:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828381.542237:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828381.542265:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280754688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828381.542275:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280754560:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828381.542291:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828381.542298:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828381.542300:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828381.542307:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2931 00000100:00080000:2.0:1787828381.542311:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828381.542315:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828381.542322:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828381.542352:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280754816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828381.542373:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828381.542382:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787828381.542383:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828381.542388:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828381.542397:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2931 00000100:00080000:2.0:1787828381.542403:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828381.542409:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828381.542417:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828381.542434:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280754944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828381.542447:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828381.542453:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828381.542461:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828381/real 1787828381] req@ffff8ab3f0439a40 x1874671280754560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828381.542462:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828381.542478:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2931 00000100:00080000:2.0:1787828381.542489:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828381.542501:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828381.542509:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828381.542516:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280754560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828381.542532:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280755072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828381.542537:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828381.542558:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828381.542573:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280754560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828381.542608:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280754688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828381.542638:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828381.542654:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828381/real 1787828381] req@ffff8ab3f043a1c0 x1874671280754688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828381.542670:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280754688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828381.542680:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828381.542684:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828381.542689:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280754688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828381.542696:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280754816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828381.542741:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828381.542763:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828381/real 1787828381] req@ffff8ab3f043a940 x1874671280754816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828381.542799:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280754816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828381.542817:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828381.542824:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828381.542830:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280754816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828381.542842:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280754944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828381.542951:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828381.542977:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828381/real 1787828381] req@ffff8ab3f043b480 x1874671280754944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828381.542993:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280754944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828381.543005:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828381.543009:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828381.543013:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280754944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828381.543021:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280755072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828381.543050:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828381.543062:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828381/real 1787828381] req@ffff8ab3f0439e00 x1874671280755072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828381.543075:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280755072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828381.543088:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828381.543091:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828381.543095:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280755072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828382.566139:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828383.590134:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828384.614127:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828385.638045:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828386.661967:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828386.661975:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828386.661979:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828386.661987:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2936 00000100:00080000:2.0:1787828386.661997:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828386.662004:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828386.662037:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280755200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828386.662064:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828386.662068:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828386.662069:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828386.662075:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2936 00000100:00080000:2.0:1787828386.662079:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828386.662082:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828386.662092:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00010000:00010000:3.0:1787828386.662095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828386.662104:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280755328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828386.662114:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828386.662117:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828386.662119:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828386.662121:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2936 00000100:00080000:2.0:1787828386.662123:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828386.662126:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828386.662129:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828386.662137:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280755456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828386.662145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828386.662146:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280755200:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828386.662149:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828386.662153:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828386.662158:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2936 00000100:00080000:2.0:1787828386.662160:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828386.662163:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828386.662167:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828386.662175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280755584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828386.662181:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828386.662184:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828386.662185:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828386.662188:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2936 00000100:00080000:2.0:1787828386.662199:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828386.662201:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828386.662205:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:0.0:1787828386.662209:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828386.662216:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280755712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828386.662253:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828386/real 1787828386] req@ffff8ab3f043b840 x1874671280755200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828386.662267:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280755200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828386.662275:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828386.662278:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828386.662281:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280755200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828386.662286:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280755328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828386.662300:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828386.662307:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828386/real 1787828386] req@ffff8ab3f0439e00 x1874671280755328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828386.662315:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280755328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828386.662320:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828386.662322:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828386.662323:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280755328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828386.662343:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280755456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828386.662354:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828386.662365:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828386/real 1787828386] req@ffff8ab3f043b480 x1874671280755456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828386.662373:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280755456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828386.662378:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828386.662380:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828386.662382:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280755456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828386.662385:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280755584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828386.662395:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828386.662400:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828386/real 1787828386] req@ffff8ab3f043a940 x1874671280755584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828386.662407:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280755584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828386.662413:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828386.662415:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828386.662417:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280755584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828386.662420:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280755712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828386.662430:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828386.662435:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828386/real 1787828386] req@ffff8ab3f043a1c0 x1874671280755712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828386.662442:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280755712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828386.662448:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828386.662450:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828386.662452:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280755712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828387.686146:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828388.710110:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828389.734167:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828390.758104:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828391.782047:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828391.782060:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828391.782066:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828391.782078:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2941 00000100:00080000:0.0:1787828391.782092:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828391.782102:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828391.782110:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828391.782163:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280755840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828391.782203:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828391.782208:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828391.782210:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828391.782215:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2941 00000100:00080000:0.0:1787828391.782219:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828391.782224:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828391.782237:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828391.782254:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280755968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828391.782280:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280755840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828391.782301:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828391.782306:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828391.782309:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828391.782318:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2941 00000100:00080000:0.0:1787828391.782325:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828391.782343:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828391.782353:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828391.782396:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828391.782401:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280756096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828391.782428:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828391.782436:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828391.782441:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828391.782450:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2941 00000100:00080000:0.0:1787828391.782457:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828391.782460:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828391/real 1787828391] req@ffff8ab3f075cb40 x1874671280755840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828391.782469:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828391.782486:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828391.782500:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280755840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828391.782511:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280756224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828391.782524:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828391.782532:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828391.782545:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280755840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828391.782554:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828391.782561:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828391.782565:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828391.782572:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2941 00000100:00080000:0.0:1787828391.782579:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828391.782587:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280755968:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787828391.782589:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828391.782604:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828391.782628:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280756352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828391.782643:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828391.782672:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828391/real 1787828391] req@ffff8ab3f075ed00 x1874671280755968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828391.782694:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280755968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828391.782715:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828391.782719:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828391.782724:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280755968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828391.782732:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280756096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828391.782751:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828391.782762:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828391/real 1787828391] req@ffff8ab3f075e940 x1874671280756096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828391.782776:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280756096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828391.782786:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828391.782789:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828391.782793:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280756096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828391.782799:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280756224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828391.782817:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828391.782828:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828391/real 1787828391] req@ffff8ab3f075f480 x1874671280756224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828391.782841:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280756224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828391.782920:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828391.782925:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828391.782930:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280756224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828391.782959:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280756352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828391.782988:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828391.783003:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828391/real 1787828391] req@ffff8ab3f075da40 x1874671280756352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828391.783028:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280756352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828391.783041:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828391.783045:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828391.783049:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280756352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828392.806148:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828393.830020:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828394.854073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828395.878060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828396.901979:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828396.901988:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828396.901992:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828396.902000:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2946 00000100:00080000:0.0:1787828396.902010:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828396.902017:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828396.902073:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280756480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828396.902081:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828396.902101:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828396.902120:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828396.902121:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828396.902125:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2946 00000100:00080000:0.0:1787828396.902129:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828396.902133:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828396.902143:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828396.902151:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280756480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828396.902158:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280756608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828396.902169:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828396.902174:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828396.902175:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828396.902179:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2946 00000100:00080000:0.0:1787828396.902182:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828396.902185:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828396.902190:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828396.902198:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280756736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828396.902216:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828396.902219:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828396.902223:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828396.902223:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828396.902228:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2946 00000100:00080000:0.0:1787828396.902233:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828396.902236:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828396.902241:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828396.902250:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280756864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828396.902257:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828396.902261:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828396.902262:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828396.902266:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2946 00000100:00080000:0.0:1787828396.902269:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828396.902274:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828396/real 1787828396] req@ffff8ab3f075e580 x1874671280756480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828396.902275:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828396.902282:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828396.902309:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280756992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828396.902319:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280756480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828396.902333:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828396.902338:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828396.902343:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280756480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828396.902362:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280756608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828396.902408:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828396.902418:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828396/real 1787828396] req@ffff8ab3f075da40 x1874671280756608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828396.902428:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280756608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828396.902440:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828396.902442:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828396.902445:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280756608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828396.902449:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280756736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828396.902463:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828396.902474:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828396/real 1787828396] req@ffff8ab3f075f480 x1874671280756736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828396.902513:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280756736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828396.902522:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828396.902526:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828396.902530:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280756736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828396.902535:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280756864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828396.902552:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828396.902560:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828396/real 1787828396] req@ffff8ab3f075e940 x1874671280756864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828396.902567:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280756864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828396.902574:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828396.902592:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828396.902603:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280756864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828396.902611:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280756992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828396.902634:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828396.902664:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828396/real 1787828396] req@ffff8ab3f075ed00 x1874671280756992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828396.902686:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280756992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828396.902693:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828396.902695:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828396.902697:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280756992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828397.926174:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828398.950079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828399.974003:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828400.998044:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828402.022041:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828402.022051:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828402.022055:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828402.022066:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2951 00000100:00080000:0.0:1787828402.022077:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828402.022085:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828402.022127:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280757120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828402.022130:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828402.022161:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828402.022165:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828402.022167:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828402.022171:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2951 00000100:00080000:0.0:1787828402.022174:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828402.022177:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828402.022186:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828402.022199:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280757248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828402.022210:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828402.022213:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828402.022215:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828402.022218:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2951 00000100:00080000:0.0:1787828402.022222:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828402.022245:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280757120:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828402.022252:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828402.022266:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828402.022293:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280757376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828402.022315:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828402.022321:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828402.022325:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828402.022333:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2951 00000100:00080000:0.0:1787828402.022342:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828402.022351:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828402.022353:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828402.022366:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828402.022389:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280757504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828402.022401:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828402.022405:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828402.022409:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828402.022416:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2951 00000100:00080000:0.0:1787828402.022421:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828402.022428:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828402.022431:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828402/real 1787828402] req@ffff8ab3f075cb40 x1874671280757120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828402.022437:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828402.022466:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280757632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828402.022477:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280757120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828402.022548:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828402.022561:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828402.022572:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280757120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828402.022621:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280757248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828402.022663:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828402.022682:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828402/real 1787828402] req@ffff8ab3f075f0c0 x1874671280757248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828402.022695:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280757248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828402.022704:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828402.022707:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828402.022711:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280757248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828402.022719:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280757376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828402.022741:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828402.022755:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828402/real 1787828402] req@ffff8ab3f075de00 x1874671280757376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828402.022766:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280757376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828402.022790:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828402.022796:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828402.022800:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280757376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828402.022808:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280757504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828402.022849:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828402.022873:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828402/real 1787828402] req@ffff8ab3f075f840 x1874671280757504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828402.022932:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280757504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828402.022944:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828402.022957:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828402.022961:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280757504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828402.022970:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280757632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828402.022992:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828402.023001:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828402/real 1787828402] req@ffff8ab3f075cf00 x1874671280757632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828402.023011:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280757632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828402.023021:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828402.023024:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828402.023026:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280757632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828403.046031:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828404.070131:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828405.094122:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828406.118161:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828407.141978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828407.141990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828407.141995:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828407.142007:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2956 00000100:00080000:0.0:1787828407.142046:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1787828407.142056:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828407.142057:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828407.142105:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280757760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828407.142139:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828407.142144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828407.142147:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828407.142152:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2956 00000100:00080000:0.0:1787828407.142157:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828407.142188:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828407.142199:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828407.142208:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280757760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828407.142220:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280757888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828407.142235:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828407.142240:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828407.142243:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828407.142248:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2956 00000100:00080000:0.0:1787828407.142271:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828407.142276:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828407.142284:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828407.142285:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828407.142302:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280758016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828407.142327:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828407.142331:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828407.142334:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828407.142339:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2956 00000100:00080000:0.0:1787828407.142345:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828407.142350:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828407.142357:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828407.142358:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828407/real 1787828407] req@ffff8ab3f075c000 x1874671280757760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828407.142374:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280758144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828407.142388:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280757760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828407.142389:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828407.142401:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828407.142405:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828407.142410:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828407.142414:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2956 00000100:00080000:1.0:1787828407.142419:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787828407.142421:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828407.142433:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828407.142434:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280757760:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787828407.142441:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828407.142455:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280758272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828407.142475:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280757888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828407.142516:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828407.142528:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828407/real 1787828407] req@ffff8ab3f075cf00 x1874671280757888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828407.142539:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280757888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828407.142546:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828407.142549:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828407.142552:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280757888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828407.142557:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280758016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828407.142578:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828407.142585:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828407/real 1787828407] req@ffff8ab3f075f840 x1874671280758016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828407.142594:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280758016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828407.142600:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828407.142603:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828407.142605:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280758016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828407.142610:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280758144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828407.142622:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828407.142634:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828407/real 1787828407] req@ffff8ab3f075de00 x1874671280758144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828407.142655:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280758144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828407.142666:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828407.142670:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828407.142674:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280758144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828407.142682:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280758272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828407.142704:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828407.142719:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828407/real 1787828407] req@ffff8ab3f075f0c0 x1874671280758272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828407.142733:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280758272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828407.142763:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828407.142767:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828407.142771:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280758272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828408.166194:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828409.190081:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828410.214117:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828411.238090:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828412.262031:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828412.262044:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828412.262050:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828412.262065:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2961 00000100:00080000:0.0:1787828412.262078:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828412.262088:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828412.262133:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828412.262156:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280758400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828412.262183:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828412.262187:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828412.262190:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828412.262194:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2961 00000100:00080000:0.0:1787828412.262199:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828412.262205:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828412.262237:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828412.262240:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280758400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828412.262299:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280758528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828412.262322:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828412.262328:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828412.262332:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828412.262339:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2961 00000800:00000100:1.0:1787828412.262362:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828412.262364:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828412.262372:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828412.262379:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828412.262390:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280758656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828412.262404:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828412.262406:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828412.262408:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828412.262412:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2961 00000100:00080000:0.0:1787828412.262415:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828412.262418:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828412/real 1787828412] req@ffff8ab3f075cb40 x1874671280758400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828412.262421:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828412.262425:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828412.262433:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280758784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828412.262446:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828412.262448:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828412.262452:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280758400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828412.262453:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828412.262461:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2961 00000100:00080000:0.0:1787828412.262467:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828412.262472:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787828412.262472:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828412.262475:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787828412.262479:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828412.262484:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280758400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828412.262489:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280758912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828412.262509:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280758528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828412.262529:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828412.262539:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828412/real 1787828412] req@ffff8ab3f075ed00 x1874671280758528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828412.262548:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280758528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828412.262554:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828412.262557:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828412.262559:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280758528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828412.262563:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280758656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828412.262573:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828412.262579:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828412/real 1787828412] req@ffff8ab3f075e940 x1874671280758656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828412.262587:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280758656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828412.262592:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828412.262594:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828412.262596:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280758656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828412.262600:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280758784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828412.262638:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828412.262650:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828412/real 1787828412] req@ffff8ab3f075f480 x1874671280758784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828412.262663:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280758784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828412.262675:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828412.262681:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828412.262687:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280758784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828412.262700:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280758912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828412.262770:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828412.262783:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828412/real 1787828412] req@ffff8ab3f075da40 x1874671280758912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828412.262795:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280758912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828412.262806:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828412.262826:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828412.262832:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280758912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828413.286030:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828414.310085:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828415.334197:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828416.358045:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828417.382035:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828417.382050:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828417.382056:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828417.382070:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2967 00000100:00080000:0.0:1787828417.382088:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828417.382102:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828417.382141:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828417.382176:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280759040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828417.382225:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828417.382232:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828417.382236:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828417.382244:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2967 00000100:00080000:0.0:1787828417.382253:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828417.382263:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828417.382281:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828417.382304:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280759040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828417.382317:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280759168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828417.382336:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828417.382341:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828417.382344:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828417.382349:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2967 00000100:00080000:0.0:1787828417.382381:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828417.382387:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828417.382394:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828417.382411:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280759296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828417.382424:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828417.382441:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828417.382443:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828417.382449:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2967 00000100:00080000:0.0:1787828417.382453:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828417.382472:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828417.382479:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828417.382502:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280759168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828417.382503:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280759424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828417.382532:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828417.382537:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828417.382541:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828417.382548:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2967 00000100:00080000:0.0:1787828417.382554:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828417.382559:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828417.382567:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828417.382571:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280759296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828417.382592:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280759552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828417.382621:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280759424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828417.382694:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280759552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828418.406108:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828419.430153:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828420.454346:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787828420.454378:0:10942:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:1.0:1787828420.454441:0:10942:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828420.454467:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787828420.454526:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787828420.454535:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787828420.454543:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787828420.454568:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:0.0:1787828420.454606:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828417/real 1787828420] req@ffff8ab3f075e940 x1874671280759424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828420.454643:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280759424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828420.454683:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828420.454696:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828420.454703:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280759424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828420.454713:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828417/real 1787828420] req@ffff8ab3f075ed00 x1874671280759552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828420.454731:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280759552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828420.454762:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828420.454767:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828420.454771:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280759552:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828420.454780:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828417/real 1787828420] req@ffff8ab3f075da40 x1874671280759168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828420.454797:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280759168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828420.454808:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828420.454813:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828420.454818:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280759168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828420.454826:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828417/real 1787828420] req@ffff8ab3f075f480 x1874671280759296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828420.454843:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280759296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828420.454872:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828420.454877:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828420.454882:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280759296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828420.454940:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828417/real 1787828420] req@ffff8ab3f075e580 x1874671280759040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828420.454958:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280759040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828420.454969:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828420.454973:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828420.454978:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280759040:192.168.203.124@tcp:250:kworker.0 00010000:00010000:3.0:1787828421.478174:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828422.502119:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828422.502144:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828422.502151:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828422.502166:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2972 00000100:00080000:0.0:1787828422.502182:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828422.502196:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828422.502202:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828422.502268:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280759680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828422.502297:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828422.502301:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828422.502303:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828422.502306:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2972 00000100:00080000:0.0:1787828422.502310:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828422.502314:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828422.502324:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828422.502338:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280759808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828422.502360:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828422.502364:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828422.502366:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828422.502373:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2972 00000100:00100000:1.0:1787828422.502375:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280759680:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828422.502376:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828422.502386:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828422.502415:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828422.502432:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280759936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828422.502445:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828422.502449:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828422.502452:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828422.502456:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2972 00000100:00080000:0.0:1787828422.502486:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828422.502512:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828422.502521:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828422.502524:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828422.502550:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280760064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828422.502565:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828422.502570:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828422.502573:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828422.502581:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2972 00000100:00080000:0.0:1787828422.502589:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828422.502590:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828422/real 1787828422] req@ffff8ab3f075e580 x1874671280759680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828422.502604:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828422.502619:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828422.502651:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280759680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828422.502668:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280760192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828422.502674:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828422.502688:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828422.502700:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280759680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828422.502740:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280759808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828422.502787:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828422.502809:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828422/real 1787828422] req@ffff8ab3f075f480 x1874671280759808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828422.502828:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280759808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828422.502841:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828422.502846:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828422.502852:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280759808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828422.502862:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280759936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828422.502930:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828422.502949:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828422/real 1787828422] req@ffff8ab3f075da40 x1874671280759936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828422.502968:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280759936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828422.502980:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828422.502985:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828422.502991:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280759936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828422.503001:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280760064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828422.503038:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828422.503060:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828422/real 1787828422] req@ffff8ab3f075ed00 x1874671280760064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828422.503082:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280760064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828422.503098:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828422.503103:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828422.503108:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280760064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828422.503118:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280760192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828422.503150:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828422.503168:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828422/real 1787828422] req@ffff8ab3f075e940 x1874671280760192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828422.503187:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280760192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828422.503204:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828422.503209:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828422.503214:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280760192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828423.526091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828424.550060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828425.574081:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828426.598081:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828427.622061:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828427.622073:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828427.622078:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828427.622089:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2977 00000100:00080000:3.0:1787828427.622100:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828427.622108:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828427.622134:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828427.622153:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280760320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828427.622184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828427.622188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828427.622190:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828427.622194:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2977 00000100:00080000:3.0:1787828427.622197:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828427.622210:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828427.622221:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828427.622245:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280760320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828427.622254:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280760448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828427.622277:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828427.622282:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828427.622285:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828427.622292:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2977 00000100:00080000:3.0:1787828427.622319:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828427.622331:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828427.622332:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787828427.622345:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828427.622401:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280760576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828427.622420:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828427.622436:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828427/real 1787828427] req@ffff8ab3d0a430c0 x1874671280760320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828427.622436:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828427.622442:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828427.622452:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2977 00000100:00080000:3.0:1787828427.622463:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828427.622473:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828427.622482:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828427.622488:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280760320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828427.622513:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828427.622513:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280760704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828427.622524:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828427.622536:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280760320:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828427.622542:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828427.622547:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828427.622550:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828427.622556:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2977 00000100:00080000:3.0:1787828427.622560:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828427.622564:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828427.622569:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828427.622572:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280760448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787828427.622586:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280760832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828427.622652:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828427.622669:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828427/real 1787828427] req@ffff8ab3d0a41e00 x1874671280760448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828427.622682:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280760448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828427.622699:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828427.622702:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828427.622705:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280760448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828427.622712:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280760576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828427.622748:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828427.622770:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828427/real 1787828427] req@ffff8ab3d0a43480 x1874671280760576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828427.622786:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280760576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828427.622795:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828427.622799:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828427.622802:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280760576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828427.622808:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280760704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828427.622831:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828427.622840:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828427/real 1787828427] req@ffff8ab3d0a43840 x1874671280760704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828427.622851:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280760704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828427.622860:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828427.622863:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828427.622866:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280760704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828427.622882:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280760832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828427.622932:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828427.622944:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828427/real 1787828427] req@ffff8ab3d0a40000 x1874671280760832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828427.622968:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280760832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828427.622977:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828427.622980:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828427.622983:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280760832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828428.646072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828429.670164:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828430.694093:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828431.718135:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00020000:3.0:1787828432.230056:0:47650:0:(osc_cache.c:999:osc_extent_wait()) extent ffff8ab3c779f540@{[0 -> 10/1023], [3|0|+|rpc|wiuY|ffff8ab3c78f5c00], [69632|11|+|-|ffff8ab3c367ac00|1024|ffff8ab4020d0000]} lustre-OST0001-osc-ffff8ab3c584d000: wait ext to 0 timedout, recovery in progress? 00000008:00020000:2.0:1787828432.230149:0:47651:0:(osc_cache.c:999:osc_extent_wait()) extent ffff8ab3c779f840@{[0 -> 38/1023], [3|0|+|rpc|wiuY|ffff8ab3c78f4a80], [184320|39|+|-|ffff8ab3c367bc00|1024|ffff8ab4020d0000]} lustre-OST0000-osc-ffff8ab3c584d000: wait ext to 0 timedout, recovery in progress? 00000008:00020000:2.0:1787828432.230169:0:47651:0:(osc_cache.c:999:osc_extent_wait()) ### extent: 000000007ba4ef22 ns: lustre-OST0000-osc-ffff8ab3c584d000 lock: 0000000022463a61/0x268e1a602cc1903 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800029400020000 nid: local remote: 0x772721ad1f0d81c3 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00020000:3.0:1787828432.269743:0:47650:0:(osc_cache.c:999:osc_extent_wait()) ### extent: 00000000289d3f60 ns: lustre-OST0001-osc-ffff8ab3c584d000 lock: 00000000f76f7717/0x268e1a602cc196c lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800029400020000 nid: local remote: 0x772721ad1f0d82d4 expref: -99 pid: 48428 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00080000:1.0:1787828432.742034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828432.742042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828432.742062:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828432.742073:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2982 00000100:00080000:1.0:1787828432.742083:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828432.742090:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828432.742143:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280760960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828432.742165:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828432.742168:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828432.742169:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828432.742172:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2982 00000100:00080000:1.0:1787828432.742174:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828432.742177:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828432.742185:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828432.742194:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280761088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828432.742202:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828432.742204:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280760960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787828432.742205:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828432.742210:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828432.742215:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2982 00000100:00080000:1.0:1787828432.742217:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828432.742219:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828432.742222:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828432.742231:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280761216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828432.742238:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828432.742240:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828432.742242:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828432.742245:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2982 00000100:00080000:1.0:1787828432.742255:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828432.742258:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828432.742262:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787828432.742264:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828432.742275:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280761344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828432.742282:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828432.742285:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828432.742286:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828432.742288:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2982 00000100:00080000:1.0:1787828432.742291:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828432.742293:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828432.742297:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828432.742306:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280761472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828432.742311:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828432/real 1787828432] req@ffff8ab3d22fc780 x1874671280760960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828432.742328:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280760960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828432.742337:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828432.742340:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828432.742344:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280760960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828432.742363:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280761088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828432.742380:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00010000:00010000:2.0:1787828432.742385:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828432.742392:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828432/real 1787828432] req@ffff8ab3d22fe1c0 x1874671280761088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828432.742401:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280761088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828432.742407:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828432.742409:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828432.742412:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280761088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828432.742416:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280761216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828432.742426:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828432.742432:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828432/real 1787828432] req@ffff8ab3d22fde00 x1874671280761216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828432.742439:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280761216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828432.742444:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828432.742445:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828432.742447:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280761216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828432.742450:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280761344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828432.742460:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828432.742466:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828432/real 1787828432] req@ffff8ab3d22fd2c0 x1874671280761344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828432.742473:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280761344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828432.742479:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828432.742481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828432.742483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280761344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828432.742486:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280761472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828432.742496:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828432.742501:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828432/real 1787828432] req@ffff8ab3d22fed00 x1874671280761472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828432.742508:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280761472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828432.742515:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828432.742516:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828432.742518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280761472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828433.766196:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828434.790105:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828435.814124:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828436.838190:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828437.862090:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828437.862104:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828437.862110:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828437.862125:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2987 00010000:00010000:3.0:1787828437.862150:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828437.862154:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828437.862172:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828437.862234:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280761600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828437.862280:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828437.862285:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828437.862288:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828437.862293:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2987 00000100:00080000:2.0:1787828437.862299:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828437.862306:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828437.862319:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828437.862346:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280761728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828437.862349:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280761600:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828437.862378:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828437.862383:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828437.862386:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828437.862392:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2987 00000100:00080000:2.0:1787828437.862397:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828437.862402:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828437.862408:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828437.862423:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280761856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828437.862436:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828437.862446:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828437.862448:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828437.862455:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828437.862463:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2987 00000100:00080000:2.0:1787828437.862470:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828437.862476:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828437.862484:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828437.862499:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280761984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828437.862512:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828437.862520:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828437/real 1787828437] req@ffff8ab3f0439a40 x1874671280761600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828437.862521:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828437.862562:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828437.862580:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2987 00000100:00080000:2.0:1787828437.862588:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828437.862595:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280761600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828437.862598:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828437.862614:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:0.0:1787828437.862623:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828437.862633:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828437.862640:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280762112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828437.862645:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280761600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828437.862696:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280761728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828437.862734:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828437.862752:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828437/real 1787828437] req@ffff8ab3f043ad00 x1874671280761728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828437.862777:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280761728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828437.862789:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828437.862795:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828437.862802:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280761728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828437.862815:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280761856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828437.862850:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828437.862874:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828437/real 1787828437] req@ffff8ab3f04392c0 x1874671280761856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828437.862968:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280761856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828437.862986:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828437.862994:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828437.863001:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280761856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828437.863014:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280761984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828437.863059:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828437.863082:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828437/real 1787828437] req@ffff8ab3f04383c0 x1874671280761984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828437.863107:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280761984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828437.863141:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828437.863147:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828437.863155:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280761984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828437.863164:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280762112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828437.863193:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828437.863222:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828437/real 1787828437] req@ffff8ab3f043bc00 x1874671280762112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828437.863237:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280762112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828437.863249:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828437.863253:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828437.863257:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280762112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828438.886068:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828439.910139:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828440.934096:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828441.958130:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828442.982020:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828442.982033:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828442.982040:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828442.982053:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2992 00000100:00080000:2.0:1787828442.982066:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828442.982076:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828442.982127:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280762240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828442.982166:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1787828442.982171:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828442.982175:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828442.982183:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828442.982194:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2992 00000100:00080000:2.0:1787828442.982202:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828442.982210:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828442.982227:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828442.982252:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280762368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828442.982270:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280762240:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828442.982272:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828442.982278:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828442.982282:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828442.982289:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2992 00000100:00080000:2.0:1787828442.982295:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828442.982301:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828442.982310:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828442.982328:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280762496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828442.982345:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828442.982350:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828442.982354:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828442.982356:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828442.982362:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2992 00000100:00080000:2.0:1787828442.982368:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828442.982374:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828442.982382:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828442.982400:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280762624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828442.982417:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828442.982423:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828442.982426:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828442/real 1787828442] req@ffff8ab3f0439680 x1874671280762240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828442.982427:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828442.982434:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2992 00000100:00080000:2.0:1787828442.982441:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828442.982448:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828442.982456:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828442.982457:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280762240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828442.982472:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280762752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828442.982477:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828442.982485:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828442.982523:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280762240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828442.982564:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280762368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828442.982608:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828442.982628:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828442/real 1787828442] req@ffff8ab3f043a580 x1874671280762368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828442.982645:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280762368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828442.982656:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828442.982660:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828442.982665:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280762368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828442.982672:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280762496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828442.982693:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828442.982704:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828442/real 1787828442] req@ffff8ab3f0438b40 x1874671280762496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828442.982718:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280762496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828442.982728:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828442.982731:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828442.982735:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280762496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828442.982741:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280762624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828442.982760:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828442.982771:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828442/real 1787828442] req@ffff8ab3f0438780 x1874671280762624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828442.982785:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280762624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828442.982796:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828442.982800:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828442.982804:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280762624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828442.982810:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280762752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828442.982827:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828442.982838:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828442/real 1787828442] req@ffff8ab3f043b0c0 x1874671280762752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828442.982852:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280762752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828442.982863:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828442.982866:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828442.982869:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280762752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828444.006108:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828445.030074:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828446.054157:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828447.078049:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828448.102054:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828448.102064:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1787828448.102067:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828448.102069:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828448.102091:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 2997 00000100:00080000:0.0:1787828448.102113:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828448.102120:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828448.102158:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280762880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828448.102185:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828448.102188:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828448.102190:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828448.102193:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2997 00000100:00080000:0.0:1787828448.102197:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828448.102202:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828448.102214:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828448.102252:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280762880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828448.102254:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280763008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828448.102278:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828448.102284:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828448.102288:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828448.102295:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2997 00000100:00080000:0.0:1787828448.102341:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828448.102346:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828448.102352:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828448.102353:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828448.102378:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280763136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828448.102405:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828448.102408:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828448.102411:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828448.102417:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2997 00000100:00080000:0.0:1787828448.102423:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828448.102428:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828448.102434:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828448.102436:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828448/real 1787828448] req@ffff8ab3f075cb40 x1874671280762880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828448.102450:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280763264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828448.102499:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828448.102502:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280762880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828448.102503:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828448.102507:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828448.102518:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 2997 00000100:00080000:0.0:1787828448.102525:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828448.102526:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828448.102531:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787828448.102537:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000008:00100000:0.0:1787828448.102540:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828448.102544:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280762880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828448.102556:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280763392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828448.102572:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280763008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828448.102597:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828448.102609:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828448/real 1787828448] req@ffff8ab3f075f0c0 x1874671280763008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828448.102620:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280763008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828448.102626:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828448.102629:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828448.102632:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280763008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828448.102636:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280763136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828448.102648:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828448.102655:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828448/real 1787828448] req@ffff8ab3f075de00 x1874671280763136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828448.102663:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280763136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828448.102670:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828448.102672:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828448.102674:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280763136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828448.102678:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280763264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828448.102689:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828448.102696:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828448/real 1787828448] req@ffff8ab3f075f840 x1874671280763264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828448.102716:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280763264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828448.102723:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828448.102725:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828448.102727:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280763264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828448.102740:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280763392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828448.102752:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828448.102759:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828448/real 1787828448] req@ffff8ab3f075cf00 x1874671280763392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828448.102787:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280763392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828448.102796:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828448.102798:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828448.102801:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280763392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828449.126229:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828450.150083:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828451.174151:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828452.198230:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828453.222143:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1787828453.222146:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828453.222164:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828453.222173:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828453.222190:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3002 00000100:00080000:0.0:1787828453.222206:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828453.222218:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828453.222277:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280763520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828453.222327:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828453.222334:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828453.222338:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828453.222346:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3002 00000100:00080000:0.0:1787828453.222355:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828453.222365:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828453.222382:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828453.222392:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280763520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828453.222441:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280763648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828453.222470:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828453.222478:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828453.222482:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828453.222492:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3002 00000100:00080000:0.0:1787828453.222500:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828453.222513:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828453.222517:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828453.222525:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828453.222555:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280763776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828453.222585:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828453.222592:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828453.222596:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828453.222605:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3002 00000100:00080000:0.0:1787828453.222616:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828453.222617:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828453/real 1787828453] req@ffff8ab3f075c000 x1874671280763520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828453.222634:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828453.222654:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828453.222679:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280763520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828453.222687:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280763904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828453.222739:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828453.222743:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828453.222752:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787828453.222757:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828453.222764:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280763520:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828453.222767:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828453.222784:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3002 00000100:00080000:0.0:1787828453.222793:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828453.222802:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828453.222809:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280763648:192.168.203.124@tcp:38:kworker.0 00000008:00100000:0.0:1787828453.222819:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828453.222852:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280764032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828453.222873:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828453.222957:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828453/real 1787828453] req@ffff8ab3f075cf00 x1874671280763648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828453.222990:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280763648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828453.223002:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828453.223007:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828453.223013:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280763648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828453.223025:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280763776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828453.223060:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828453.223080:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828453/real 1787828453] req@ffff8ab3f075f840 x1874671280763776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828453.223109:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280763776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828453.223120:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828453.223124:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828453.223129:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280763776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828453.223137:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280763904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828453.223157:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828453.223176:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828453/real 1787828453] req@ffff8ab3f075de00 x1874671280763904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828453.223189:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280763904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828453.223201:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828453.223204:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828453.223208:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280763904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828453.223230:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280764032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828453.223249:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828453.223261:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828453/real 1787828453] req@ffff8ab3f075f0c0 x1874671280764032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828453.223275:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280764032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828453.223286:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828453.223289:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828453.223293:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280764032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828454.246159:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828455.270132:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828456.294089:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828457.318151:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828458.342028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828458.342042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828458.342049:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828458.342063:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3007 00000100:00080000:0.0:1787828458.342086:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828458.342096:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828458.342158:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280764160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828458.342175:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828458.342203:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828458.342220:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828458.342239:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828458.342250:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3007 00000100:00080000:0.0:1787828458.342260:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828458.342270:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828458.342282:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280764160:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787828458.342302:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828458.342330:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280764288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828458.342349:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828458.342353:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828458.342356:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828458.342363:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3007 00000100:00080000:0.0:1787828458.342400:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828458.342408:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828458.342415:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828458.342425:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828458.342440:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280764416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828458.342467:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828458.342471:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828458.342475:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828458.342480:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3007 00000100:00080000:0.0:1787828458.342487:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828458.342494:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828458.342502:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828458.342511:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828458/real 1787828458] req@ffff8ab3f075cb40 x1874671280764160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828458.342521:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280764544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828458.342564:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828458.342567:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280764160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828458.342573:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828458.342580:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828458.342595:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3007 00000100:00080000:1.0:1787828458.342607:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828458.342619:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828458.342625:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787828458.342632:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828458.342641:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828458.342645:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280764160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828458.342659:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280764672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828458.342708:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280764288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828458.342747:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828458.342777:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828458/real 1787828458] req@ffff8ab3f075e940 x1874671280764288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828458.342794:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280764288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828458.342805:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828458.342809:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828458.342814:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280764288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828458.342821:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280764416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828458.342839:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828458.342849:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828458/real 1787828458] req@ffff8ab3f075ed00 x1874671280764416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828458.342863:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280764416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828458.342881:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828458.342919:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828458.342924:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280764416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828458.342943:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280764544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828458.342966:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828458.342979:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828458/real 1787828458] req@ffff8ab3f075da40 x1874671280764544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828458.342994:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280764544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828458.343016:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828458.343020:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828458.343024:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280764544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828458.343030:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280764672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828458.343049:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828458.343060:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828458/real 1787828458] req@ffff8ab3f075f480 x1874671280764672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828458.343082:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280764672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828458.343092:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828458.343096:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828458.343100:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280764672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828459.366136:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828460.390218:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828461.413990:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828462.438078:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828463.462004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828463.462014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828463.462018:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828463.462029:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3013 00000100:00080000:0.0:1787828463.462039:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828463.462047:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828463.462060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828463.462094:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280764800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828463.462118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828463.462121:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828463.462123:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828463.462127:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3013 00000100:00080000:0.0:1787828463.462130:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828463.462133:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828463.462144:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828463.462159:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280764928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828463.462161:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280764800:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828463.462184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828463.462187:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828463.462189:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828463.462194:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3013 00000100:00080000:0.0:1787828463.462197:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828463.462200:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828463.462205:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828463.462234:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280765056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828463.462244:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828463.462248:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828463.462253:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828463.462254:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828463.462262:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3013 00000100:00080000:0.0:1787828463.462269:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828463.462274:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828463.462281:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828463.462293:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280765184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828463.462302:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828463.462305:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828463.462307:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828463.462311:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3013 00000100:00080000:0.0:1787828463.462315:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828463.462324:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828463.462325:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828463/real 1787828463] req@ffff8ab3f075e580 x1874671280764800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828463.462332:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828463.462385:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280765312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828463.462392:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280764800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828463.462407:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828463.462411:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828463.462417:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280764800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828463.462460:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280764928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828463.462487:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828463.462502:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828463/real 1787828463] req@ffff8ab3f075f480 x1874671280764928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828463.462516:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280764928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828463.462525:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828463.462528:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828463.462532:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280764928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828463.462539:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280765056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828463.462555:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828463.462565:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828463/real 1787828463] req@ffff8ab3f075da40 x1874671280765056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828463.462577:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280765056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828463.462586:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828463.462589:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828463.462592:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280765056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828463.462598:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280765184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828463.462613:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828463.462622:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828463/real 1787828463] req@ffff8ab3f075ed00 x1874671280765184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828463.462634:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280765184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828463.462644:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828463.462647:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828463.462650:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280765184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828463.462655:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280765312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828463.462669:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828463.462678:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828463/real 1787828463] req@ffff8ab3f075e940 x1874671280765312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828463.462707:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280765312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828463.462724:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828463.462745:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828463.462750:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280765312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828464.486128:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828465.510040:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828466.534079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828467.558027:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828468.582029:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828468.582038:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828468.582042:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828468.582051:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3018 00000100:00080000:2.0:1787828468.582060:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828468.582066:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828468.582103:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280765440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828468.582128:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828468.582135:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828468.582140:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828468.582141:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828468.582145:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3018 00000100:00080000:2.0:1787828468.582148:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828468.582151:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828468.582160:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828468.582174:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280765440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828468.582174:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280765568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828468.582189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828468.582192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828468.582193:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828468.582197:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3018 00000100:00080000:2.0:1787828468.582199:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828468.582201:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828468.582205:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828468.582214:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280765696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828468.582221:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828468.582224:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828468.582225:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828468.582229:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3018 00000100:00080000:2.0:1787828468.582232:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828468.582235:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828468.582242:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787828468.582244:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828468.582255:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280765824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828468.582262:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828468.582265:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828468.582266:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828468.582269:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3018 00000100:00080000:2.0:1787828468.582271:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828468.582275:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828468.582279:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828468.582289:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828468/real 1787828468] req@ffff8ab3f0438f00 x1874671280765440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828468.582291:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280765952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828468.582365:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280765440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828468.582374:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828468.582377:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828468.582380:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280765440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828468.582399:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280765568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828468.582417:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828468.582426:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828468/real 1787828468] req@ffff8ab3f043b0c0 x1874671280765568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828468.582435:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280765568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828468.582441:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828468.582443:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828468.582445:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280765568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828468.582450:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280765696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828468.582460:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828468.582466:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828468/real 1787828468] req@ffff8ab3f0438780 x1874671280765696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828468.582473:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280765696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828468.582479:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828468.582481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828468.582483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280765696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828468.582486:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280765824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828468.582496:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828468.582502:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828468/real 1787828468] req@ffff8ab3f0438b40 x1874671280765824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828468.582509:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280765824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828468.582524:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828468.582526:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828468.582529:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280765824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828468.582533:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280765952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828468.582545:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828468.582551:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828468/real 1787828468] req@ffff8ab3f043a580 x1874671280765952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828468.582559:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280765952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828468.582566:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828468.582567:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828468.582569:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280765952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828469.606082:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828470.630133:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828471.654083:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828472.678065:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828473.701982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828473.701992:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828473.701996:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828473.702024:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3023 00000100:00080000:2.0:1787828473.702035:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828473.702043:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828473.702088:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828473.702090:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280766080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828473.702118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828473.702122:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828473.702124:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828473.702127:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3023 00000100:00080000:2.0:1787828473.702130:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828473.702134:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828473.702144:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828473.702175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280766208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828473.702186:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828473.702190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828473.702192:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828473.702196:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3023 00000100:00080000:2.0:1787828473.702200:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828473.702207:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828473.702208:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280766080:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787828473.702214:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828473.702230:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280766336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828473.702241:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828473.702245:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828473.702247:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828473.702251:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3023 00000100:00080000:2.0:1787828473.702255:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828473.702259:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828473.702264:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828473.702315:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280766464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828473.702324:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828473.702328:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828473.702330:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828473.702337:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828473.702340:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3023 00000100:00080000:2.0:1787828473.702347:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828473.702351:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828473.702357:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828473.702369:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280766592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828473.702409:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828473/real 1787828473] req@ffff8ab3f0439680 x1874671280766080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828473.702437:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280766080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828473.702453:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828473.702459:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828473.702465:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280766080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828473.702476:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280766208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828473.702509:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828473.702546:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828473/real 1787828473] req@ffff8ab3f043bc00 x1874671280766208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828473.702567:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280766208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828473.702590:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828473.702595:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828473.702600:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280766208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828473.702632:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280766336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828473.702664:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828473.702681:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828473/real 1787828473] req@ffff8ab3f04383c0 x1874671280766336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828473.702700:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280766336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828473.702712:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828473.702717:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828473.702723:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280766336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828473.702732:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280766464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828473.702762:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828473.702780:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828473/real 1787828473] req@ffff8ab3f04392c0 x1874671280766464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828473.702817:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280766464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828473.702832:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828473.702838:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828473.702843:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280766464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828473.702853:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280766592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828473.702882:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828473.702950:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828473/real 1787828473] req@ffff8ab3f043ad00 x1874671280766592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828473.702969:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280766592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828473.702983:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828473.702988:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828473.702992:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280766592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828474.726243:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828475.750112:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828476.774204:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828477.798171:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828478.821967:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828478.821981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828478.821986:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828478.822000:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3028 00000100:00080000:2.0:1787828478.822013:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828478.822024:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828478.822066:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280766720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828478.822079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828478.822094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828478.822102:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828478.822104:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828478.822108:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3028 00000100:00080000:2.0:1787828478.822113:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828478.822117:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828478.822127:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828478.822144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280766848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828478.822145:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280766720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828478.822165:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828478.822169:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828478.822171:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828478.822175:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3028 00000100:00080000:2.0:1787828478.822178:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828478.822181:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828478.822186:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828478.822197:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280766976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828478.822206:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828478.822209:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828478.822211:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828478.822216:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3028 00000800:00000100:1.0:1787828478.822222:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828478.822222:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828478.822231:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828478.822237:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828478.822266:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280767104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828478.822275:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828478.822278:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828478.822280:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828478.822285:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3028 00000100:00080000:2.0:1787828478.822290:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828478.822295:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828478.822301:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828478/real 1787828478] req@ffff8ab3f0439a40 x1874671280766720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828478.822302:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828478.822319:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280767232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828478.822329:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280766720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828478.822341:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828478.822347:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828478.822351:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280766720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828478.822375:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280766848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828478.822397:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828478.822408:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828478/real 1787828478] req@ffff8ab3f043a1c0 x1874671280766848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828478.822419:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280766848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828478.822427:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828478.822430:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828478.822433:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280766848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828478.822438:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280766976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828478.822450:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828478.822458:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828478/real 1787828478] req@ffff8ab3f043a940 x1874671280766976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828478.822467:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280766976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828478.822474:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828478.822476:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828478.822478:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280766976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828478.822483:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280767104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828478.822495:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828478.822503:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828478/real 1787828478] req@ffff8ab3f043b480 x1874671280767104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828478.822512:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280767104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828478.822521:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828478.822523:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828478.822525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280767104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828478.822530:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280767232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828478.822542:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828478.822548:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828478/real 1787828478] req@ffff8ab3f0439e00 x1874671280767232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828478.822557:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280767232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828478.822565:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828478.822567:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828478.822570:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280767232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828479.846154:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828480.870178:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828481.894029:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828482.918066:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828483.942046:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828483.942051:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828483.942058:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828483.942062:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828483.942071:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3033 00000100:00080000:0.0:1787828483.942080:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828483.942086:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828483.942120:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280767360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828483.942145:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828483.942148:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828483.942149:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828483.942152:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3033 00000100:00080000:0.0:1787828483.942155:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828483.942158:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828483.942167:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828483.942177:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280767488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828483.942192:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280767360:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828483.942195:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828483.942218:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828483.942221:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828483.942225:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3033 00000100:00080000:0.0:1787828483.942227:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828483.942231:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828483.942235:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828483.942247:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280767616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828483.942267:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828483.942270:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828483.942271:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828483.942274:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3033 00000100:00080000:0.0:1787828483.942279:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828483.942282:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828483.942287:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828483.942296:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280767744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828483.942303:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828483.942306:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828483.942307:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828483.942310:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3033 00000100:00080000:0.0:1787828483.942314:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828483.942318:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828483.942323:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828483.942333:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280767872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828483.942341:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280767488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828483.942373:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280767616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828483.942390:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280767744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828483.942415:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280767872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828484.966266:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828485.990166:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828487.014254:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787828487.014311:0:10944:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787828487.014345:0:10944:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828487.014359:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787828487.014419:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787828487.014430:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787828487.014441:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787828487.014452:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787828487.014477:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828483/real 1787828487] req@ffff8ab3f075cf00 x1874671280767872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828487.014512:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280767872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828487.014528:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828487.014533:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828487.014538:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280767872:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828487.014546:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828483/real 1787828487] req@ffff8ab3f075f0c0 x1874671280767488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828487.014560:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280767488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828487.014570:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828487.014574:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828487.014578:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280767488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828487.014585:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828483/real 1787828487] req@ffff8ab3f075de00 x1874671280767616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828487.014598:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280767616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828487.014607:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828487.014611:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828487.014615:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280767616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828487.014622:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828483/real 1787828487] req@ffff8ab3f075f840 x1874671280767744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828487.014643:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280767744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828487.014654:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828487.014658:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828487.014661:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280767744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828487.014668:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828483/real 1787828487] req@ffff8ab3f075cb40 x1874671280767360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828487.014687:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280767360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828487.014696:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828487.014700:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828487.014704:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280767360:192.168.203.124@tcp:250:kworker.0 00010000:00010000:2.0:1787828488.038149:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828489.062087:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828489.062101:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828489.062107:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828489.062122:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3038 00000100:00080000:0.0:1787828489.062136:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828489.062147:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828489.062198:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828489.062229:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280768000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828489.062266:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828489.062272:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828489.062274:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828489.062279:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3038 00000100:00080000:0.0:1787828489.062285:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828489.062290:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828489.062302:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828489.062322:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280768128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828489.062338:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828489.062347:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828489.062353:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828489.062356:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280768000:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828489.062361:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3038 00000100:00080000:0.0:1787828489.062369:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828489.062378:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828489.062385:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828489.062413:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280768256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828489.062426:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828489.062431:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828489.062433:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828489.062441:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3038 00000100:00080000:0.0:1787828489.062463:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828489.062472:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828489.062484:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828489.062486:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828489.062523:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280768384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828489.062542:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828489.062549:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828489.062552:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828489.062562:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3038 00000100:00100000:1.0:1787828489.062572:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828489/real 1787828489] req@ffff8ab3f075c000 x1874671280768000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828489.062575:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828489.062594:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828489.062615:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828489.062630:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280768000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828489.062644:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280768512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828489.062670:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828489.062686:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828489.062697:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280768000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828489.062744:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280768128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828489.062819:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828489.062849:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828489/real 1787828489] req@ffff8ab3f075cb40 x1874671280768128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828489.062876:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280768128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828489.062936:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828489.062944:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828489.062952:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280768128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828489.062967:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280768256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828489.063011:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828489.063038:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828489/real 1787828489] req@ffff8ab3f075f840 x1874671280768256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828489.063062:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280768256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828489.063079:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828489.063086:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828489.063104:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280768256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828489.063118:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280768384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828489.063160:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828489.063210:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828489/real 1787828489] req@ffff8ab3f075de00 x1874671280768384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828489.063236:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280768384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828489.063254:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828489.063260:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828489.063267:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280768384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828489.063278:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280768512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828489.063331:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828489.063356:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828489/real 1787828489] req@ffff8ab3f075f0c0 x1874671280768512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828489.063380:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280768512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828489.063399:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828489.063406:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828489.063411:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280768512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828490.086147:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828491.110028:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828492.134089:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828493.158029:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828494.182053:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828494.182066:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828494.182072:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828494.182084:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3043 00000100:00080000:0.0:1787828494.182098:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828494.182109:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828494.182162:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828494.182196:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280768640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828494.182244:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828494.182251:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828494.182254:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828494.182262:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3043 00000100:00080000:0.0:1787828494.182281:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828494.182290:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828494.182305:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828494.182325:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280768640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828494.182337:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280768768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828494.182364:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828494.182371:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828494.182375:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828494.182383:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3043 00000100:00080000:0.0:1787828494.182413:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828494.182422:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828494.182428:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828494.182439:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828494.182461:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280768896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828494.182476:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828494.182480:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828494.182482:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828494.182489:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3043 00000100:00080000:0.0:1787828494.182497:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828494.182506:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828494/real 1787828494] req@ffff8ab3f075cf00 x1874671280768640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828494.182508:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828494.182525:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828494.182565:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280768640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828494.182573:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280769024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828494.182588:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828494.182601:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828494.182611:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280768640:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828494.182612:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828494.182627:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828494.182632:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828494.182641:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3043 00000100:00080000:0.0:1787828494.182649:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828494.182652:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280768768:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787828494.182665:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828494.182677:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787828494.182699:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828494.182710:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280769152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828494.182731:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828494/real 1787828494] req@ffff8ab3f075e940 x1874671280768768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828494.182761:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280768768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828494.182772:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828494.182776:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828494.182780:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280768768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828494.182787:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280768896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828494.182809:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828494.182818:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828494/real 1787828494] req@ffff8ab3f075ed00 x1874671280768896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828494.182828:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280768896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828494.182835:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828494.182838:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828494.182841:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280768896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828494.182847:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280769024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828494.182871:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828494.182885:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828494/real 1787828494] req@ffff8ab3f075da40 x1874671280769024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828494.182933:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280769024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828494.182943:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828494.182946:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828494.182949:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280769024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828494.182965:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280769152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828494.182986:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828494.182995:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828494/real 1787828494] req@ffff8ab3f075f480 x1874671280769152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828494.183004:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280769152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828494.183012:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828494.183014:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828494.183017:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280769152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828495.206115:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828496.230013:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828497.254268:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828498.278017:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828499.302069:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828499.302080:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828499.302084:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828499.302093:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3048 00000100:00080000:2.0:1787828499.302146:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828499.302152:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828499.302216:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280769280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828499.302231:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828499.302240:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828499.302245:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828499.302262:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828499.302265:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3048 00000100:00080000:2.0:1787828499.302278:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828499.302280:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828499.302288:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828499.302303:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280769408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828499.302315:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828499.302316:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280769280:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828499.302323:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828499.302327:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828499.302332:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3048 00000100:00080000:2.0:1787828499.302345:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828499.302350:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828499.302355:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828499.302367:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280769536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828499.302375:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828499.302377:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828499.302379:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828499.302381:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3048 00000100:00080000:2.0:1787828499.302386:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828499.302389:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828499.302392:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828499.302409:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828499.302412:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280769664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828499.302421:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828499.302423:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828499.302425:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828499.302428:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3048 00000100:00080000:2.0:1787828499.302430:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828499.302433:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828499.302437:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828499.302444:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280769792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828499.302466:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828499/real 1787828499] req@ffff8ab3f043b840 x1874671280769280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828499.302492:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280769280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828499.302500:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828499.302504:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828499.302507:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280769280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828499.302525:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280769408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828499.302542:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828499.302561:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828499/real 1787828499] req@ffff8ab3f0439e00 x1874671280769408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828499.302570:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280769408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828499.302579:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828499.302583:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828499.302587:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280769408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828499.302593:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280769536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828499.302614:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828499.302626:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828499/real 1787828499] req@ffff8ab3f043b480 x1874671280769536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828499.302636:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280769536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828499.302647:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828499.302649:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828499.302652:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280769536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828499.302656:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280769664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828499.302670:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828499.302686:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828499/real 1787828499] req@ffff8ab3f043a940 x1874671280769664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828499.302698:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280769664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828499.302707:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828499.302711:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828499.302714:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280769664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828499.302720:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280769792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828499.302746:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828499.302757:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828499/real 1787828499] req@ffff8ab3f043a1c0 x1874671280769792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828499.302770:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280769792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828499.302780:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828499.302784:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828499.302788:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280769792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828500.326029:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828501.350108:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828502.374048:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828503.398157:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828504.422085:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828504.422101:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828504.422107:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828504.422148:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3054 00000100:00080000:2.0:1787828504.422159:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828504.422168:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828504.422222:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280769920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828504.422233:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828504.422258:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828504.422264:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828504.422266:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828504.422271:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3054 00000100:00080000:2.0:1787828504.422275:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828504.422279:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828504.422290:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828504.422306:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280770048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828504.422320:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828504.422323:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280769920:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828504.422325:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828504.422332:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828504.422345:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3054 00000100:00080000:2.0:1787828504.422349:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828504.422354:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828504.422358:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828504.422373:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280770176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828504.422383:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828504.422387:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828504.422388:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828504.422402:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3054 00000100:00080000:2.0:1787828504.422415:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828504.422420:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828504.422426:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828504.422431:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828504.422445:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280770304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828504.422457:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828504.422461:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828504.422463:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828504.422468:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3054 00000100:00080000:2.0:1787828504.422471:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828504.422477:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828504.422483:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828504.422495:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828504/real 1787828504] req@ffff8ab3f0439a40 x1874671280769920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828504.422500:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280770432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828504.422568:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280769920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828504.422582:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828504.422587:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828504.422592:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280769920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828504.422618:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280770048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828504.422641:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828504.422655:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828504/real 1787828504] req@ffff8ab3f043ad00 x1874671280770048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828504.422668:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280770048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828504.422677:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828504.422681:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828504.422685:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280770048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828504.422690:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280770176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828504.422706:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828504.422732:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828504/real 1787828504] req@ffff8ab3f04392c0 x1874671280770176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828504.422744:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280770176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828504.422752:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828504.422755:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828504.422758:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280770176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828504.422764:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280770304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828504.422782:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828504.422792:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828504/real 1787828504] req@ffff8ab3f04383c0 x1874671280770304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828504.422804:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280770304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828504.422813:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828504.422816:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828504.422819:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280770304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828504.422831:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280770432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828504.422846:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828504.422855:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828504/real 1787828504] req@ffff8ab3f043bc00 x1874671280770432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828504.422890:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280770432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828504.422949:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828504.422957:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828504.422964:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280770432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828505.446201:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828506.470174:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828507.494162:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828508.518077:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828509.542110:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828509.542123:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828509.542129:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828509.542143:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3059 00000100:00080000:2.0:1787828509.542157:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828509.542168:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828509.542221:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828509.542222:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280770560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828509.542268:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828509.542273:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828509.542276:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828509.542280:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3059 00000100:00080000:2.0:1787828509.542285:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828509.542290:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828509.542303:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828509.542319:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280770688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828509.542335:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828509.542349:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280770560:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828509.542350:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828509.542363:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828509.542386:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3059 00000100:00080000:2.0:1787828509.542391:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828509.542396:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828509.542402:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828509.542423:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280770816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828509.542435:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828509.542441:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828509.542444:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828509.542451:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3059 00000100:00080000:2.0:1787828509.542491:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828509.542498:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828509.542505:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828509.542510:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828509.542528:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280770944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828509.542546:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828509.542551:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828509.542553:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828509.542559:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3059 00000100:00080000:2.0:1787828509.542563:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828509.542569:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828509.542582:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828509.542610:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828509/real 1787828509] req@ffff8ab3f0439680 x1874671280770560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828509.542620:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280771072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828509.542665:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280770560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828509.542688:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828509.542697:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828509.542708:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280770560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828509.542749:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280770688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828509.542787:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828509.542813:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828509/real 1787828509] req@ffff8ab3f043a580 x1874671280770688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828509.542840:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280770688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828509.542859:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828509.542867:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828509.542874:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280770688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828509.542926:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280770816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828509.542959:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828509.542976:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828509/real 1787828509] req@ffff8ab3f0438b40 x1874671280770816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828509.542992:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280770816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828509.543003:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828509.543008:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828509.543012:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280770816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828509.543020:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280770944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828509.543075:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828509.543090:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828509/real 1787828509] req@ffff8ab3f0438780 x1874671280770944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828509.543105:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280770944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828509.543119:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828509.543124:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828509.543128:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280770944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828509.543136:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280771072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828509.543157:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828509.543168:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828509/real 1787828509] req@ffff8ab3f043b0c0 x1874671280771072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828509.543182:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280771072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828509.543195:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828509.543198:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828509.543202:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280771072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828510.566117:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828511.590164:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828512.614186:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828513.638121:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828514.662068:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828514.662086:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828514.662093:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828514.662110:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3064 00000100:00080000:2.0:1787828514.662123:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828514.662134:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828514.662138:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828514.662184:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280771200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828514.662228:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828514.662233:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828514.662236:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828514.662240:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3064 00000100:00080000:2.0:1787828514.662245:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828514.662249:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828514.662262:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828514.662280:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280771328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828514.662294:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828514.662299:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828514.662302:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828514.662307:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3064 00000100:00080000:2.0:1787828514.662311:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828514.662317:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280771200:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828514.662320:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828514.662330:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828514.662349:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280771456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828514.662363:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828514.662368:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828514.662370:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828514.662376:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3064 00000100:00080000:2.0:1787828514.662396:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828514.662402:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828514.662408:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828514.662417:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828514.662440:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280771584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828514.662454:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828514.662459:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828514.662461:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828514.662465:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3064 00000100:00080000:2.0:1787828514.662470:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828514.662475:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828514.662480:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828514.662498:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280771712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828514.662520:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828514/real 1787828514] req@ffff8ab3f0438f00 x1874671280771200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828514.662559:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280771200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828514.662592:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828514.662600:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828514.662607:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280771200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828514.662619:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280771328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828514.662654:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828514.662673:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828514/real 1787828514] req@ffff8ab3f043b0c0 x1874671280771328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828514.662689:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280771328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828514.662709:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828514.662716:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828514.662723:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280771328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828514.662764:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280771456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828514.662803:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828514.662834:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828514/real 1787828514] req@ffff8ab3f0438780 x1874671280771456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828514.662852:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280771456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828514.662869:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828514.662925:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828514.662933:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280771456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828514.662947:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280771584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828514.662980:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828514.663027:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828514/real 1787828514] req@ffff8ab3f0438b40 x1874671280771584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828514.663044:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280771584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828514.663057:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828514.663061:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828514.663066:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280771584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828514.663074:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280771712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828514.663100:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828514.663112:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828514/real 1787828514] req@ffff8ab3f043a580 x1874671280771712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828514.663126:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280771712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828514.663138:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828514.663141:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828514.663145:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280771712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828515.686118:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828516.710186:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828517.734283:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828518.758220:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828519.782021:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828519.782031:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828519.782036:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828519.782046:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3069 00000100:00080000:2.0:1787828519.782055:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828519.782062:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828519.782081:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828519.782101:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280771840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828519.782127:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828519.782130:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828519.782132:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828519.782135:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3069 00000100:00080000:2.0:1787828519.782138:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828519.782141:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828519.782150:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828519.782167:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280771968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828519.782168:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280771840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828519.782209:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828519.782213:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828519.782215:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828519.782219:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3069 00000100:00080000:2.0:1787828519.782222:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828519.782227:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828519.782233:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828519.782244:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280772096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828519.782259:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828519.782265:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787828519.782267:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828519.782268:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828519.782276:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3069 00000100:00080000:2.0:1787828519.782280:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828519.782284:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828519.782289:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828519.782300:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280772224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828519.782308:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828519.782312:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828519.782314:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828519.782318:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3069 00000100:00080000:2.0:1787828519.782322:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828519.782326:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828519.782331:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828519.782332:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828519/real 1787828519] req@ffff8ab3f0439680 x1874671280771840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828519.782347:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280772352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828519.782358:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280771840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828519.782370:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828519.782375:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828519.782380:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280771840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828519.782406:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280771968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828519.782436:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828519.782456:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828519/real 1787828519] req@ffff8ab3f043bc00 x1874671280771968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828519.782474:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280771968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828519.782483:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828519.782488:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828519.782494:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280771968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828519.782503:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280772096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828519.782546:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828519.782561:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828519/real 1787828519] req@ffff8ab3f04383c0 x1874671280772096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828519.782577:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280772096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828519.782584:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828519.782587:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828519.782589:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280772096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828519.782594:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280772224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828519.782631:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828519.782641:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828519/real 1787828519] req@ffff8ab3f04392c0 x1874671280772224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828519.782651:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280772224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828519.782660:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828519.782662:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828519.782665:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280772224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828519.782670:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280772352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828519.782682:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828519.782689:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828519/real 1787828519] req@ffff8ab3f043ad00 x1874671280772352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828519.782698:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280772352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828519.782706:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828519.782709:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828519.782711:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280772352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828520.806215:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828521.830146:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828522.854174:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828523.878135:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828524.902025:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828524.902037:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828524.902042:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828524.902056:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3074 00000100:00080000:2.0:1787828524.902068:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828524.902078:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828524.902130:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280772480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828524.902130:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828524.902162:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828524.902166:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828524.902168:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828524.902172:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3074 00000100:00080000:2.0:1787828524.902177:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828524.902184:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828524.902198:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828524.902218:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280772480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828524.902218:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280772608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828524.902259:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828524.902265:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828524.902267:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828524.902273:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3074 00000100:00080000:2.0:1787828524.902276:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828524.902280:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828524.902287:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828524.902305:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280772736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828524.902323:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828524.902328:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828524.902330:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828524.902335:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3074 00000100:00080000:2.0:1787828524.902342:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828524.902347:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828524.902355:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828524.902361:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828524.902374:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280772864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828524.902389:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828524.902393:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828524.902395:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828524.902408:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3074 00000100:00080000:2.0:1787828524.902415:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828524.902419:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828524.902424:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828524.902435:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280772992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828524.902474:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828524/real 1787828524] req@ffff8ab3f0439a40 x1874671280772480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828524.902498:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280772480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828524.902510:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828524.902515:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828524.902520:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280772480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828524.902553:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280772608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828524.902593:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828524.902604:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828524/real 1787828524] req@ffff8ab3f043a1c0 x1874671280772608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828524.902614:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280772608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828524.902621:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828524.902624:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828524.902626:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280772608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828524.902631:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280772736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828524.902662:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828524.902671:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828524/real 1787828524] req@ffff8ab3f043a940 x1874671280772736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828524.902681:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280772736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828524.902689:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828524.902692:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828524.902694:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280772736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828524.902700:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280772864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828524.902741:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828524.902759:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828524/real 1787828524] req@ffff8ab3f043b480 x1874671280772864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828524.902795:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280772864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828524.902814:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828524.902818:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828524.902821:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280772864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828524.902829:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280772992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828524.902854:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828524.902866:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828524/real 1787828524] req@ffff8ab3f0439e00 x1874671280772992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828524.902879:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280772992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828524.902952:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828524.902956:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828524.902959:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280772992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828525.926128:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828526.950115:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828527.974105:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828528.998110:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828530.022095:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828530.022113:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828530.022135:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828530.022148:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3079 00000100:00080000:0.0:1787828530.022159:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828530.022166:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828530.022188:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828530.022203:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280773120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828530.022229:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828530.022233:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828530.022234:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828530.022237:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3079 00000100:00080000:0.0:1787828530.022240:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828530.022244:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828530.022269:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828530.022290:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280773120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828530.022291:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280773248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828530.022316:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828530.022320:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828530.022322:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828530.022327:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3079 00000100:00080000:0.0:1787828530.022331:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828530.022335:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828530.022344:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828530.022364:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280773376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828530.022382:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828530.022388:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828530.022391:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828530.022401:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828530.022409:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3079 00000100:00080000:0.0:1787828530.022421:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828530.022431:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828530.022440:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828530.022480:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828530/real 1787828530] req@ffff8ab3f075e580 x1874671280773120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828530.022482:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280773504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828530.022517:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828530.022520:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828530.022521:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828530.022530:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3079 00000100:00080000:0.0:1787828530.022537:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828530.022543:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828530.022548:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828530.022561:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280773120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828530.022571:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280773632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828530.022576:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828530.022587:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828530.022594:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280773120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828530.022626:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280773248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828530.022656:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828530.022673:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828530/real 1787828530] req@ffff8ab3f075f480 x1874671280773248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828530.022688:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280773248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828530.022697:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828530.022701:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828530.022705:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280773248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828530.022712:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280773376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828530.022732:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828530.022744:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828530/real 1787828530] req@ffff8ab3f075da40 x1874671280773376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828530.022778:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280773376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828530.022792:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828530.022797:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828530.022803:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280773376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828530.022813:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280773504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828530.022840:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828530.022855:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828530/real 1787828530] req@ffff8ab3f075ed00 x1874671280773504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828530.022874:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280773504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828530.022988:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828530.022996:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828530.023004:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280773504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828530.023018:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280773632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828530.023073:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828530.023089:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828530/real 1787828530] req@ffff8ab3f075e940 x1874671280773632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828530.023104:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280773632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828530.023116:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828530.023119:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828530.023124:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280773632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828531.046084:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828532.070110:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828533.094150:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828534.118027:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828535.142019:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828535.142031:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828535.142037:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828535.142048:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3084 00000100:00080000:0.0:1787828535.142060:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828535.142070:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828535.142075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828535.142136:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280773760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828535.142173:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828535.142179:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828535.142182:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828535.142189:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3084 00000100:00080000:0.0:1787828535.142212:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828535.142218:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828535.142228:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828535.142231:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280773760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828535.142247:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280773888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828535.142260:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828535.142264:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828535.142266:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828535.142270:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3084 00000100:00080000:0.0:1787828535.142288:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828535.142292:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828535.142297:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828535.142301:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828535.142314:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280774016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828535.142332:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828535.142335:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828535.142337:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828535.142343:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3084 00000100:00080000:0.0:1787828535.142348:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828535.142353:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828535.142357:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828535/real 1787828535] req@ffff8ab3f075cf00 x1874671280773760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828535.142361:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828535.142379:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280774144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828535.142387:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280773760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828535.142395:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828535.142407:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828535.142413:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828535.142417:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828535.142422:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3084 00000100:00080000:1.0:1787828535.142428:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787828535.142431:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828535.142436:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280773760:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828535.142441:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828535.142449:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828535.142460:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280774272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828535.142469:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280773888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828535.142495:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828535.142508:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828535/real 1787828535] req@ffff8ab3f075f0c0 x1874671280773888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828535.142521:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280773888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828535.142545:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828535.142548:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828535.142555:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280773888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828535.142561:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280774016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828535.142579:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828535.142591:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828535/real 1787828535] req@ffff8ab3f075de00 x1874671280774016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828535.142605:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280774016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828535.142621:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828535.142625:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828535.142628:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280774016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828535.142633:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280774144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828535.142649:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828535.142656:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828535/real 1787828535] req@ffff8ab3f075f840 x1874671280774144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828535.142666:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280774144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828535.142675:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828535.142677:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828535.142680:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280774144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828535.142693:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280774272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828535.142707:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828535.142715:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828535/real 1787828535] req@ffff8ab3f075cb40 x1874671280774272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828535.142725:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280774272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828535.142744:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828535.142746:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828535.142749:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280774272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828536.166079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828537.190118:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828538.214156:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828539.238026:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828540.262018:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828540.262028:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828540.262033:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828540.262044:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3089 00000100:00080000:0.0:1787828540.262055:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828540.262063:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828540.262111:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828540.262117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280774400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828540.262144:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828540.262147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828540.262149:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828540.262152:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3089 00000100:00080000:0.0:1787828540.262155:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828540.262159:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828540.262169:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828540.262203:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280774528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828540.262211:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280774400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828540.262218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828540.262225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828540.262227:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828540.262232:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3089 00000100:00080000:0.0:1787828540.262235:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828540.262239:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828540.262244:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828540.262267:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280774656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828540.262277:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828540.262281:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828540.262283:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828540.262300:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3089 00000100:00080000:0.0:1787828540.262304:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828540.262324:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828540.262329:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828540.262332:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828540.262345:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280774784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828540.262357:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828540.262360:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828540.262362:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828540.262366:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3089 00000100:00080000:0.0:1787828540.262370:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828540.262392:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828540.262396:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828540.262405:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828540/real 1787828540] req@ffff8ab3f075c000 x1874671280774400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828540.262410:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280774912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828540.262465:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280774400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828540.262483:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828540.262490:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828540.262496:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280774400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828540.262526:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280774528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828540.262552:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828540.262583:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828540/real 1787828540] req@ffff8ab3f075cb40 x1874671280774528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828540.262594:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280774528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828540.262603:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828540.262608:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828540.262612:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280774528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828540.262620:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280774656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828540.262665:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828540.262683:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828540/real 1787828540] req@ffff8ab3f075f840 x1874671280774656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828540.262701:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280774656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828540.262717:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828540.262720:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828540.262723:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280774656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828540.262729:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280774784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828540.262750:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828540.262760:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828540/real 1787828540] req@ffff8ab3f075de00 x1874671280774784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828540.262771:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280774784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828540.262780:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828540.262783:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828540.262786:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280774784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828540.262791:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280774912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828540.262805:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828540.262849:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828540/real 1787828540] req@ffff8ab3f075f0c0 x1874671280774912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828540.262861:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280774912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828540.262880:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828540.262883:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828540.262886:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280774912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828541.286067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828542.310097:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828543.334044:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828544.358204:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828545.381968:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828545.381979:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828545.381984:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828545.381994:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3095 00000100:00080000:0.0:1787828545.382006:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828545.382013:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828545.382050:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280775040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828545.382085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828545.382088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828545.382090:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828545.382094:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3095 00000100:00080000:0.0:1787828545.382097:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828545.382100:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828545.382109:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828545.382140:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280775168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828545.382154:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828545.382161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828545.382164:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828545.382200:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3095 00000100:00080000:0.0:1787828545.382207:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828545.382213:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828545.382221:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00010000:00010000:3.0:1787828545.382226:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828545.382229:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280775040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828545.382244:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280775296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828545.382261:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828545.382265:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828545.382268:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828545.382274:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3095 00000100:00080000:0.0:1787828545.382292:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828545.382298:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787828545.382304:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828545.382326:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828545.382344:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280775424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828545.382355:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828545.382359:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828545.382362:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828545.382367:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3095 00000100:00080000:0.0:1787828545.382373:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828545.382374:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828545/real 1787828545] req@ffff8ab3f075cf00 x1874671280775040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828545.382381:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828545.382394:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828545.382399:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280775040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828545.382410:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828545.382413:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828545.382420:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280775552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828545.382421:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280775040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828545.382435:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280775168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828545.382457:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828545.382467:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828545/real 1787828545] req@ffff8ab3f075e940 x1874671280775168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828545.382475:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280775168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828545.382481:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828545.382483:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828545.382485:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280775168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828545.382504:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280775296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828545.382515:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828545.382522:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828545/real 1787828545] req@ffff8ab3f075ed00 x1874671280775296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828545.382536:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280775296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828545.382541:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828545.382543:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828545.382545:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280775296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828545.382549:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280775424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828545.382560:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828545.382566:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828545/real 1787828545] req@ffff8ab3f075da40 x1874671280775424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828545.382574:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280775424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828545.382582:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828545.382583:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828545.382585:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280775424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828545.382589:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280775552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828545.382599:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828545.382605:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828545/real 1787828545] req@ffff8ab3f075f480 x1874671280775552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828545.382613:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280775552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828545.382624:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828545.382627:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828545.382629:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280775552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828546.406208:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828547.430253:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828548.454135:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828549.478173:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828550.501995:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828550.502003:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828550.502007:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828550.502016:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3100 00000100:00080000:2.0:1787828550.502025:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828550.502032:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828550.502074:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280775680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828550.502100:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828550.502103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828550.502105:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828550.502108:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3100 00000100:00080000:2.0:1787828550.502112:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828550.502116:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828550.502126:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828550.502142:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280775808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828550.502148:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280775680:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828550.502155:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828550.502159:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828550.502161:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828550.502165:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3100 00000100:00080000:2.0:1787828550.502168:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828550.502173:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828550.502180:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00010000:00010000:3.0:1787828550.502181:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828550.502198:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280775936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828550.502211:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828550.502216:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828550.502218:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828550.502232:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3100 00000100:00080000:2.0:1787828550.502237:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828550.502241:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828550.502246:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828550.502260:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280776064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828550.502265:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280775808:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787828550.502272:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828550.502276:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828550.502277:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828550.502281:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3100 00000100:00080000:2.0:1787828550.502284:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828550.502287:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828550.502290:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828550.502304:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280776192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828550.502315:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280775936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828550.502347:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280776064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828550.502370:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280776192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828551.526346:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828552.550105:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828553.574115:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828553.574172:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787828553.574208:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828553.574221:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828553.574251:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828553.574257:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828553.574264:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828553.574273:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:2.0:1787828553.574299:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828550/real 1787828553] req@ffff8ab3f043a940 x1874671280776064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828553.574331:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280776064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828553.574346:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828553.574351:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828553.574356:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280776064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828553.574365:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828550/real 1787828553] req@ffff8ab3f043a1c0 x1874671280776192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828553.574378:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280776192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828553.574388:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828553.574392:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828553.574396:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280776192:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828553.574402:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828550/real 1787828553] req@ffff8ab3f0439e00 x1874671280775808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828553.574416:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280775808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828553.574425:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828553.574430:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828553.574433:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280775808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828553.574440:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828550/real 1787828553] req@ffff8ab3f043b480 x1874671280775936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828553.574453:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280775936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828553.574462:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828553.574464:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828553.574468:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280775936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828553.574474:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828550/real 1787828553] req@ffff8ab3f043b840 x1874671280775680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828553.574487:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280775680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828553.574494:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828553.574497:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828553.574500:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280775680:192.168.203.124@tcp:250:kworker.0 00010000:00010000:3.0:1787828554.598217:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828555.622009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828555.622022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828555.622027:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828555.622041:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3105 00000100:00080000:2.0:1787828555.622053:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828555.622063:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828555.622115:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280776320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828555.622177:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828555.622183:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828555.622186:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828555.622191:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3105 00000100:00080000:2.0:1787828555.622198:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828555.622204:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828555.622219:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828555.622235:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280776448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828555.622253:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828555.622259:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828555.622262:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828555.622270:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3105 00000100:00080000:2.0:1787828555.622274:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828555.622279:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828555.622288:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828555.622290:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280776320:192.168.203.124@tcp:250:kworker.0 00010000:00010000:3.0:1787828555.622292:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828555.622308:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280776576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828555.622324:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828555.622329:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828555.622331:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828555.622337:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3105 00000100:00080000:2.0:1787828555.622341:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828555.622347:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828555.622354:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828555.622369:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280776704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828555.622382:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828555.622391:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828555.622396:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828555.622403:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828555.622426:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3105 00000100:00080000:2.0:1787828555.622437:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828555.622446:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828555.622456:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828555.622483:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280776832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828555.622497:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828555/real 1787828555] req@ffff8ab3f043b840 x1874671280776320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828555.622589:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280776320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828555.622613:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828555.622621:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828555.622629:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280776320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828555.622644:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280776448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828555.622692:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828555.622752:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828555/real 1787828555] req@ffff8ab3f043b480 x1874671280776448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828555.622780:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280776448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828555.622798:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828555.622805:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828555.622813:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280776448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828555.622858:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280776576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828555.622999:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828555.623027:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828555/real 1787828555] req@ffff8ab3f0439e00 x1874671280776576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828555.623053:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280776576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828555.623069:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828555.623075:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828555.623082:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280776576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828555.623097:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280776704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828555.623144:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828555.623177:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828555/real 1787828555] req@ffff8ab3f043a1c0 x1874671280776704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828555.623205:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280776704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828555.623226:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828555.623234:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828555.623241:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280776704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828555.623254:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280776832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828555.623296:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828555.623321:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828555/real 1787828555] req@ffff8ab3f043a940 x1874671280776832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828555.623345:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280776832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828555.623364:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828555.623370:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828555.623377:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280776832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828556.646398:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828557.670161:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828558.694095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828559.718132:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828560.742003:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828560.742016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828560.742023:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828560.742035:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3110 00000100:00080000:2.0:1787828560.742049:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828560.742060:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828560.742113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280776960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828560.742142:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828560.742151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828560.742159:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828560.742163:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828560.742168:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3110 00000100:00080000:2.0:1787828560.742173:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828560.742178:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828560.742191:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828560.742210:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280777088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828560.742228:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280776960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828560.742230:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828560.742246:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828560.742249:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828560.742256:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3110 00000100:00080000:2.0:1787828560.742259:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828560.742264:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828560.742270:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828560.742285:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280777216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828560.742298:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828560.742303:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828560.742306:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828560.742310:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3110 00000100:00080000:2.0:1787828560.742326:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828560.742332:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787828560.742344:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787828560.742345:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828560.742366:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280777344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828560.742381:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828560.742398:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828560.742401:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828560.742407:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3110 00000100:00080000:2.0:1787828560.742413:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828560.742418:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828560.742427:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828560/real 1787828560] req@ffff8ab3f0439a40 x1874671280776960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828560.742429:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828560.742452:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280777472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828560.742465:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280776960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828560.742484:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828560.742493:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828560.742503:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280776960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828560.742554:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280777088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828560.742639:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828560.742659:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828560/real 1787828560] req@ffff8ab3f043ad00 x1874671280777088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828560.742706:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280777088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828560.742736:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828560.742740:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828560.742745:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280777088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828560.742753:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280777216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828560.742781:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828560.742794:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828560/real 1787828560] req@ffff8ab3f04392c0 x1874671280777216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828560.742808:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280777216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828560.742819:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828560.742823:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828560.742827:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280777216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828560.742834:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280777344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828560.742852:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828560.742863:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828560/real 1787828560] req@ffff8ab3f04383c0 x1874671280777344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828560.742876:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280777344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828560.742934:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828560.742938:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828560.742944:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280777344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828560.742952:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280777472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828560.742974:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828560.742988:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828560/real 1787828560] req@ffff8ab3f043bc00 x1874671280777472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828560.743007:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280777472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828560.743019:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828560.743023:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828560.743027:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280777472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828561.766118:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828562.790086:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828563.814087:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828564.838093:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828565.862094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828565.862127:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828565.862134:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828565.862151:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3115 00000100:00080000:2.0:1787828565.862169:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828565.862183:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828565.862183:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828565.862283:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280777600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828565.862347:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828565.862352:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828565.862356:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828565.862366:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3115 00000100:00080000:2.0:1787828565.862375:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828565.862385:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828565.862406:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828565.862410:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280777600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828565.862443:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280777728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828565.862486:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828565.862490:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828565.862493:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828565.862499:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3115 00000100:00080000:2.0:1787828565.862563:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828565.862570:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787828565.862580:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787828565.862582:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828565.862611:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280777856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828565.862626:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828565.862631:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828565.862656:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828565.862667:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3115 00000100:00080000:2.0:1787828565.862674:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828565.862681:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828565.862708:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828565.862716:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828565/real 1787828565] req@ffff8ab3f0439680 x1874671280777600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828565.862731:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280777984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828565.862747:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828565.862750:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280777600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828565.862754:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828565.862761:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828565.862806:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3115 00000100:00080000:0.0:1787828565.862810:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828565.862812:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828565.862825:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:0.0:1787828565.862833:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000008:00100000:2.0:1787828565.862838:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828565.862867:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280777600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828565.862876:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280778112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828565.862967:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280777728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828565.863021:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828565.863047:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828565/real 1787828565] req@ffff8ab3f043a580 x1874671280777728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828565.863081:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280777728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828565.863097:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828565.863102:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828565.863106:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280777728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828565.863114:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280777856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828565.863141:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828565.863154:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828565/real 1787828565] req@ffff8ab3f0438b40 x1874671280777856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828565.863167:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280777856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828565.863176:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828565.863179:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828565.863183:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280777856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828565.863189:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280777984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828565.863205:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828565.863215:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828565/real 1787828565] req@ffff8ab3f0438780 x1874671280777984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828565.863227:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280777984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828565.863239:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828565.863253:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828565.863257:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280777984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828565.863262:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280778112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828565.863277:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828565.863286:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828565/real 1787828565] req@ffff8ab3f043b0c0 x1874671280778112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828565.863327:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280778112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828565.863340:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828565.863343:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828565.863347:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280778112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828566.886160:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828567.910199:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828568.934177:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828569.958179:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828570.982040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828570.982055:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828570.982061:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828570.982076:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3120 00000100:00080000:0.0:1787828570.982089:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1787828570.982097:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828570.982098:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828570.982142:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280778240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828570.982182:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828570.982187:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828570.982189:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828570.982193:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3120 00000100:00080000:0.0:1787828570.982196:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828570.982200:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828570.982210:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828570.982224:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280778368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828570.982234:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828570.982237:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828570.982239:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828570.982244:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3120 00000100:00080000:0.0:1787828570.982247:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828570.982253:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828570.982259:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828570.982268:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280778496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828570.982282:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828570.982291:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280778240:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828570.982292:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828570.982302:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828570.982314:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3120 00000100:00080000:0.0:1787828570.982318:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828570.982322:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828570.982328:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828570.982342:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280778624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828570.982356:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828570.982361:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828570.982364:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828570.982371:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3120 00000100:00080000:0.0:1787828570.982381:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828570.982389:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787828570.982396:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828570.982408:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828570.982434:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280778752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828570.982481:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828570/real 1787828570] req@ffff8ab3f075e580 x1874671280778240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828570.982502:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280778240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828570.982514:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828570.982518:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828570.982522:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280778240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828570.982529:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280778368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828570.982549:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828570.982559:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828570/real 1787828570] req@ffff8ab3f075f480 x1874671280778368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828570.982570:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280778368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828570.982577:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828570.982579:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828570.982582:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280778368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828570.982628:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280778496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828570.982643:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828570.982652:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828570/real 1787828570] req@ffff8ab3f075da40 x1874671280778496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828570.982661:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280778496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828570.982668:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828570.982671:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828570.982674:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280778496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828570.982679:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280778624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828570.982691:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828570.982698:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828570/real 1787828570] req@ffff8ab3f075ed00 x1874671280778624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828570.982707:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280778624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828570.982721:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828570.982724:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828570.982726:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280778624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828570.982731:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280778752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828570.982743:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828570.982749:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828570/real 1787828570] req@ffff8ab3f075e940 x1874671280778752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828570.982758:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280778752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828570.982766:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828570.982768:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828570.982770:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280778752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828572.006121:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828573.030056:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828574.054178:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828575.078070:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828576.102073:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828576.102085:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828576.102091:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828576.102104:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3125 00000100:00080000:1.0:1787828576.102116:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787828576.102121:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828576.102128:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828576.102181:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280778880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828576.102222:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828576.102229:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828576.102233:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828576.102241:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3125 00000100:00080000:1.0:1787828576.102247:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828576.102253:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828576.102265:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828576.102267:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280778880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828576.102292:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280779008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828576.102308:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828576.102342:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828576.102345:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828576.102352:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3125 00000100:00080000:1.0:1787828576.102371:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828576.102380:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787828576.102389:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:1.0:1787828576.102395:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828576.102419:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280779136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828576.102433:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828576.102437:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828576.102440:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828576.102444:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3125 00000100:00080000:1.0:1787828576.102465:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828576.102470:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828576.102478:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828576.102493:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828576/real 1787828576] req@ffff8ab3d22fed00 x1874671280778880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828576.102499:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280779264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828576.102546:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828576.102551:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828576.102554:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828576.102579:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3125 00000100:00100000:2.0:1787828576.102580:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280778880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828576.102588:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828576.102601:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828576.102616:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:2.0:1787828576.102618:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828576.102628:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828576.102635:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280779392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828576.102639:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280778880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828576.102693:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280779008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828576.102738:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828576.102764:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828576/real 1787828576] req@ffff8ab3d22fd2c0 x1874671280779008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828576.102781:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280779008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828576.102790:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828576.102794:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828576.102798:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280779008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828576.102806:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280779136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828576.102844:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828576.102857:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828576/real 1787828576] req@ffff8ab3d22fde00 x1874671280779136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828576.102870:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280779136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828576.102879:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828576.102882:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828576.102920:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280779136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828576.102927:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280779264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828576.102949:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828576.102961:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828576/real 1787828576] req@ffff8ab3d22fe1c0 x1874671280779264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828576.102974:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280779264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828576.102985:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828576.102988:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828576.102991:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280779264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828576.102998:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280779392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828576.103024:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828576.103033:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828576/real 1787828576] req@ffff8ab3d22fc780 x1874671280779392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828576.103044:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280779392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828576.103054:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828576.103057:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828576.103060:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280779392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828577.126107:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828578.150118:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828579.174139:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828580.198036:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828581.222070:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828581.222075:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828581.222088:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828581.222095:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828581.222111:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3130 00000100:00080000:0.0:1787828581.222127:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828581.222141:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828581.222218:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280779520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828581.222274:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828581.222280:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828581.222283:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828581.222289:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3130 00000100:00080000:0.0:1787828581.222298:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828581.222305:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828581.222334:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828581.222337:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280779520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828581.222380:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280779648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828581.222407:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828581.222414:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828581.222423:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828581.222427:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828581.222436:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3130 00000100:00080000:0.0:1787828581.222444:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828581.222451:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828581.222460:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828581.222470:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828581/real 1787828581] req@ffff8ab3f075cf00 x1874671280779520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828581.222485:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280779776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828581.222492:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280779520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828581.222549:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828581.222568:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787828581.222579:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828581.222581:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280779520:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828581.222601:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828581.222605:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828581.222615:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3130 00000100:00080000:0.0:1787828581.222622:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828581.222630:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280779648:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787828581.222640:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828581.222650:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787828581.222659:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828581.222673:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280779904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828581.222680:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828581/real 1787828581] req@ffff8ab3f075f0c0 x1874671280779648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828581.222765:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280779648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828581.222771:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828581.222786:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828581.222789:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828581.222789:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828581.222815:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3130 00000100:00080000:2.0:1787828581.222817:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787828581.222838:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828581.222840:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280779648:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787828581.222853:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828581.222859:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280779776:192.168.203.124@tcp:38:kworker.0 00000008:00100000:0.0:1787828581.222862:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:2.0:1787828581.222922:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828581.222937:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828581/real 1787828581] req@ffff8ab3f075de00 x1874671280779776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828581.222947:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280779776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828581.222959:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828581.222961:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828581.222965:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280779776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828581.222983:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280779904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828581.222997:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280780032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828581.223001:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828581.223019:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828581/real 1787828581] req@ffff8ab3f075f840 x1874671280779904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828581.223052:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280779904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828581.223060:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828581.223065:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828581.223068:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280779904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828581.223089:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280780032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828581.223113:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828581.223131:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828581/real 1787828581] req@ffff8ab3f075cb40 x1874671280780032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828581.223141:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280780032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828581.223150:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828581.223155:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828581.223164:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280780032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828582.246086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828583.270042:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828584.294012:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828585.318010:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828586.341991:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828586.341999:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828586.342003:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828586.342012:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3135 00010000:00010000:2.0:1787828586.342021:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828586.342021:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828586.342040:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828586.342074:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280780160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828586.342094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828586.342114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828586.342116:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828586.342119:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3135 00000100:00080000:3.0:1787828586.342121:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828586.342126:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280780160:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828586.342131:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828586.342140:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828586.342152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280780288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828586.342160:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828586.342162:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828586.342164:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828586.342167:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3135 00000100:00080000:3.0:1787828586.342172:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828586.342176:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828586.342179:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828586.342184:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828586.342193:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280780416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828586.342202:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828586.342204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828586.342205:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828586.342208:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3135 00000100:00080000:3.0:1787828586.342210:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828586.342214:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828586.342227:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828586.342237:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280780544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828586.342244:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828586/real 1787828586] req@ffff8ab3d0a43c00 x1874671280780160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828586.342246:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828586.342254:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828586.342258:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828586.342265:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3135 00000100:00080000:3.0:1787828586.342268:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828586.342271:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828586.342275:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280780160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828586.342276:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:2.0:1787828586.342289:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828586.342290:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280780672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828586.342297:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828586.342310:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280780160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828586.342336:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280780288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828586.342354:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828586.342363:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828586/real 1787828586] req@ffff8ab3d0a412c0 x1874671280780288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828586.342372:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280780288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828586.342378:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828586.342380:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828586.342382:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280780288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828586.342388:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280780416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828586.342398:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828586.342403:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828586/real 1787828586] req@ffff8ab3d0a421c0 x1874671280780416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828586.342411:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280780416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828586.342416:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828586.342417:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828586.342419:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280780416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828586.342431:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280780544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828586.342441:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828586.342447:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828586/real 1787828586] req@ffff8ab3d0a40000 x1874671280780544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828586.342454:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280780544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828586.342460:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828586.342462:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828586.342463:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280780544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828586.342467:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280780672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828586.342476:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828586.342481:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828586/real 1787828586] req@ffff8ab3d0a43840 x1874671280780672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828586.342489:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280780672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828586.342516:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828586.342518:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828586.342521:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280780672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828587.366063:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828588.390038:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828589.414088:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828590.438050:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828591.462005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828591.462017:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828591.462022:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828591.462034:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3141 00000100:00080000:3.0:1787828591.462046:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828591.462053:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828591.462085:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828591.462107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280780800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828591.462130:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828591.462133:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828591.462135:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828591.462138:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3141 00000100:00080000:3.0:1787828591.462141:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828591.462144:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828591.462155:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828591.462168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280780928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828591.462170:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280780800:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828591.462195:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828591.462198:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828591.462199:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828591.462204:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3141 00000100:00080000:3.0:1787828591.462208:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828591.462211:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828591.462215:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828591.462226:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280781056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828591.462233:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828591.462236:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828591.462238:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828591.462241:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3141 00000100:00080000:3.0:1787828591.462243:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828591.462246:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828591.462249:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787828591.462256:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828591.462261:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280781184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828591.462269:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828591.462273:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828591.462274:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828591.462278:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3141 00000100:00080000:3.0:1787828591.462281:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828591.462284:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828591.462288:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828591.462302:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280781312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828591.462312:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828591/real 1787828591] req@ffff8ab3d0a43480 x1874671280780800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828591.462336:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280780800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828591.462345:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828591.462348:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828591.462351:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280780800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828591.462371:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280780928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828591.462387:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828591.462394:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828591/real 1787828591] req@ffff8ab3d0a41e00 x1874671280780928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828591.462407:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280780928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828591.462413:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828591.462415:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828591.462416:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280780928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828591.462421:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280781056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828591.462431:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828591.462437:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828591/real 1787828591] req@ffff8ab3d0a430c0 x1874671280781056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828591.462444:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280781056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828591.462449:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828591.462451:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828591.462453:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280781056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828591.462474:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280781184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828591.462493:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828591.462501:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828591/real 1787828591] req@ffff8ab3d0a43840 x1874671280781184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828591.462509:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280781184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828591.462521:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828591.462523:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828591.462525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280781184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828591.462529:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280781312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828591.462540:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828591.462545:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828591/real 1787828591] req@ffff8ab3d0a40000 x1874671280781312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828591.462552:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280781312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828591.462558:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828591.462560:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828591.462562:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280781312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828592.486034:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828593.510029:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828594.534046:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828595.558127:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828596.581989:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828596.581997:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828596.582001:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828596.582009:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3146 00000100:00080000:3.0:1787828596.582018:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828596.582023:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828596.582058:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828596.582059:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280781440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828596.582076:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828596.582080:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828596.582081:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828596.582085:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3146 00000100:00080000:3.0:1787828596.582088:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828596.582090:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828596.582098:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828596.582102:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280781440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828596.582109:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280781568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828596.582118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828596.582120:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828596.582122:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828596.582125:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3146 00000100:00080000:3.0:1787828596.582127:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828596.582130:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828596.582134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828596.582142:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280781696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828596.582149:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828596.582152:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828596.582153:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828596.582156:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3146 00000100:00080000:3.0:1787828596.582158:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828596.582160:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787828596.582165:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787828596.582166:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828596.582177:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280781824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828596.582184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828596.582187:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828596.582188:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828596.582190:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3146 00000100:00080000:3.0:1787828596.582192:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828596.582195:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828596.582198:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828596.582206:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280781952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828596.582217:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828596/real 1787828596] req@ffff8ab3d0a421c0 x1874671280781440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828596.582237:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280781440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828596.582250:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828596.582254:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828596.582259:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280781440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828596.582281:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280781568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828596.582301:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828596.582314:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828596/real 1787828596] req@ffff8ab3d0a412c0 x1874671280781568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828596.582328:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280781568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828596.582338:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828596.582341:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828596.582345:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280781568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828596.582351:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280781696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828596.582367:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828596.582378:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828596/real 1787828596] req@ffff8ab3d0a43c00 x1874671280781696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828596.582392:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280781696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828596.582401:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828596.582405:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828596.582410:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280781696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828596.582416:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280781824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828596.582454:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828596.582468:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828596/real 1787828596] req@ffff8ab3d0a40000 x1874671280781824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828596.582482:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280781824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828596.582493:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828596.582496:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828596.582500:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280781824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828596.582506:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280781952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828596.582524:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828596.582536:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828596/real 1787828596] req@ffff8ab3d0a43840 x1874671280781952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828596.582549:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280781952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828596.582559:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828596.582562:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828596.582566:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280781952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828597.606057:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828598.630016:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828599.654010:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828600.678051:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828601.702021:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828601.702032:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828601.702036:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828601.702048:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3151 00010000:00010000:3.0:1787828601.702060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828601.702061:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828601.702074:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828601.702136:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280782080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828601.702167:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828601.702170:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828601.702172:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828601.702176:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3151 00000100:00080000:2.0:1787828601.702180:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828601.702186:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828601.702198:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828601.702218:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280782208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828601.702224:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280782080:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828601.702248:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828601.702252:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828601.702254:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828601.702259:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3151 00000100:00080000:2.0:1787828601.702265:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828601.702270:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828601.702277:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828601.702290:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280782336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828601.702299:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828601.702300:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828601.702309:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828601.702313:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828601.702318:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3151 00000100:00080000:2.0:1787828601.702325:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828601.702330:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828601.702333:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828601.702345:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280782464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828601.702351:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828601.702356:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828601/real 1787828601] req@ffff8ab3f0438f00 x1874671280782080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828601.702358:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828601.702360:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828601.702376:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3151 00000100:00080000:2.0:1787828601.702396:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828601.702403:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828601.702409:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280782080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828601.702412:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:0.0:1787828601.702430:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828601.702434:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280782592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828601.702440:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828601.702454:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280782080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828601.702488:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280782208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828601.702508:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828601.702516:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828601/real 1787828601] req@ffff8ab3f043b0c0 x1874671280782208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828601.702526:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280782208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828601.702532:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828601.702534:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828601.702536:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280782208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828601.702540:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280782336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828601.702550:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828601.702556:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828601/real 1787828601] req@ffff8ab3f0438780 x1874671280782336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828601.702564:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280782336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828601.702569:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828601.702572:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828601.702574:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280782336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828601.702578:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280782464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828601.702587:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828601.702593:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828601/real 1787828601] req@ffff8ab3f0438b40 x1874671280782464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828601.702600:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280782464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828601.702611:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828601.702613:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828601.702615:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280782464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828601.702619:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280782592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828601.702628:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828601.702633:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828601/real 1787828601] req@ffff8ab3f043a580 x1874671280782592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828601.702640:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280782592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828601.702646:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828601.702648:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828601.702650:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280782592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828602.726092:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828603.750030:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828604.774058:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828605.798104:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828606.822014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828606.822027:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828606.822033:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828606.822046:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3156 00000100:00080000:2.0:1787828606.822059:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828606.822068:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828606.822072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828606.822120:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280782720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828606.822160:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828606.822164:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828606.822166:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828606.822170:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3156 00000100:00080000:2.0:1787828606.822174:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828606.822178:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828606.822191:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828606.822209:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280782848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828606.822218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828606.822221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828606.822223:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828606.822226:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3156 00000100:00080000:2.0:1787828606.822229:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828606.822233:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828606.822241:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828606.822242:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280782720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828606.822259:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280782976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828606.822271:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828606.822274:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828606.822275:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828606.822279:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3156 00000100:00080000:2.0:1787828606.822282:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828606.822285:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828606.822289:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828606.822299:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280783104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828606.822306:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828606.822310:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828606.822311:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828606.822319:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828606.822323:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3156 00000100:00080000:2.0:1787828606.822328:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828606.822333:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828606.822338:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828606.822364:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280783232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828606.822395:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828606/real 1787828606] req@ffff8ab3f0439680 x1874671280782720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828606.822410:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280782720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828606.822418:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828606.822422:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828606.822425:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280782720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828606.822432:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280782848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828606.822451:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828606.822459:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828606/real 1787828606] req@ffff8ab3f043bc00 x1874671280782848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828606.822467:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280782848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828606.822472:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828606.822474:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828606.822476:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280782848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828606.822494:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280782976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828606.822505:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828606.822511:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828606/real 1787828606] req@ffff8ab3f04383c0 x1874671280782976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828606.822518:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280782976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828606.822523:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828606.822525:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828606.822527:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280782976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828606.822531:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280783104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828606.822541:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828606.822547:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828606/real 1787828606] req@ffff8ab3f04392c0 x1874671280783104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828606.822554:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280783104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828606.822561:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828606.822562:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828606.822564:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280783104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828606.822568:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280783232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828606.822577:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828606.822582:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828606/real 1787828606] req@ffff8ab3f043ad00 x1874671280783232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828606.822589:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280783232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828606.822595:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828606.822596:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828606.822598:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280783232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828607.846552:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828608.870028:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828609.894014:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828610.918007:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828611.941999:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828611.942011:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828611.942017:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828611.942028:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3161 00010000:00010000:2.0:1787828611.942045:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828611.942050:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828611.942062:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828611.942112:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280783360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828611.942141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828611.942145:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828611.942147:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828611.942151:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3161 00000100:00080000:3.0:1787828611.942155:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828611.942160:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828611.942172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828611.942187:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280783488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828611.942197:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828611.942199:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280783360:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828611.942201:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828611.942207:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828611.942211:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3161 00000100:00080000:3.0:1787828611.942214:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828611.942217:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828611.942221:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828611.942231:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280783616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828611.942238:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828611.942241:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828611.942242:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828611.942245:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3161 00000100:00080000:3.0:1787828611.942263:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828611.942266:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828611.942269:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787828611.942274:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828611.942281:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280783744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828611.942288:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828611.942292:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828611.942293:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828611.942297:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3161 00000100:00080000:3.0:1787828611.942299:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828611.942301:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828611.942305:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828611.942328:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280783872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828611.942346:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828611/real 1787828611] req@ffff8ab3d0a430c0 x1874671280783360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828611.942368:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280783360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828611.942378:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828611.942381:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828611.942385:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280783360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828611.942408:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280783488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828611.942427:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828611.942442:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828611/real 1787828611] req@ffff8ab3d0a41e00 x1874671280783488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828611.942451:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280783488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828611.942457:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828611.942460:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828611.942462:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280783488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828611.942466:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280783616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828611.942478:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828611.942484:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828611/real 1787828611] req@ffff8ab3d0a43480 x1874671280783616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828611.942492:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280783616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828611.942498:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828611.942500:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828611.942503:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280783616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828611.942506:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280783744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828611.942525:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828611.942532:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828611/real 1787828611] req@ffff8ab3d0a43840 x1874671280783744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828611.942545:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280783744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828611.942556:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828611.942559:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828611.942563:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280783744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828611.942570:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280783872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828611.942590:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828611.942598:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828611/real 1787828611] req@ffff8ab3d0a40000 x1874671280783872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828611.942607:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280783872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828611.942614:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828611.942617:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828611.942619:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280783872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828612.966075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828613.990126:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828615.014111:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828616.038042:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828617.061994:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828617.062002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828617.062006:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828617.062014:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3166 00000100:00080000:3.0:1787828617.062023:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828617.062028:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828617.062062:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280784000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828617.062067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828617.062087:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828617.062090:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828617.062092:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828617.062094:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3166 00000100:00080000:3.0:1787828617.062096:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828617.062099:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828617.062107:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828617.062115:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280784128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828617.062123:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828617.062125:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828617.062127:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828617.062130:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280784000:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828617.062134:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3166 00000100:00080000:3.0:1787828617.062140:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828617.062143:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828617.062146:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828617.062155:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280784256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828617.062161:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828617.062164:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828617.062165:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828617.062167:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3166 00000100:00080000:3.0:1787828617.062182:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828617.062184:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828617.062188:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828617.062196:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280784384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828617.062202:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828617.062204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828617.062206:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828617.062208:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3166 00000100:00080000:3.0:1787828617.062210:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828617.062212:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828617.062216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828617.062224:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280784512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828617.062243:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280784128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828617.062273:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280784256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828617.062312:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280784384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828617.062331:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280784512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828618.086095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828619.110088:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828620.134173:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828620.134201:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:3.0:1787828620.134243:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828620.134276:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787828620.134313:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787828620.134325:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787828620.134334:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787828620.134345:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:0.0:1787828620.134373:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828617/real 1787828620] req@ffff8ab3d0a412c0 x1874671280784128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828620.134401:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280784128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828620.134412:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828620.134418:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828620.134423:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280784128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828620.134430:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828617/real 1787828620] req@ffff8ab3d0a421c0 x1874671280784256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828620.134442:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280784256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828620.134450:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828620.134454:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828620.134457:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280784256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828620.134463:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828617/real 1787828620] req@ffff8ab3d0a40000 x1874671280784384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828620.134474:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280784384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828620.134485:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828620.134488:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828620.134491:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280784384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828620.134497:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828617/real 1787828620] req@ffff8ab3d0a43840 x1874671280784512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828620.134507:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280784512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828620.134516:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828620.134520:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828620.134523:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280784512:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828620.134528:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828617/real 1787828620] req@ffff8ab3d0a43c00 x1874671280784000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828620.134538:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280784000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828620.134545:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828620.134548:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828620.134551:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280784000:192.168.203.124@tcp:250:kworker.0 00010000:00010000:2.0:1787828621.158057:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828622.181985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828622.181996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828622.182001:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828622.182012:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3171 00000100:00080000:3.0:1787828622.182023:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828622.182032:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828622.182046:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828622.182075:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280784640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828622.182102:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828622.182107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828622.182109:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828622.182113:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3171 00000100:00080000:3.0:1787828622.182117:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828622.182120:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828622.182131:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828622.182144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280784768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828622.182154:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280784640:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828622.182162:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828622.182167:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828622.182170:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828622.182176:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3171 00000100:00080000:3.0:1787828622.182180:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828622.182185:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828622.182190:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828622.182203:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280784896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828622.182215:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828622.182221:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828622.182222:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828622.182227:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828622.182232:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3171 00000100:00080000:3.0:1787828622.182238:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828622.182255:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828622.182261:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828622.182275:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280785024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828622.182287:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828622/real 1787828622] req@ffff8ab3d0a43480 x1874671280784640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828622.182288:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828622.182299:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828622.182307:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828622.182318:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3171 00000100:00080000:3.0:1787828622.182323:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828622.182329:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280784640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828622.182331:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828622.182341:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:0.0:1787828622.182348:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828622.182356:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787828622.182358:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280785152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828622.182380:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280784640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828622.182407:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280784768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828622.182434:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828622.182449:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828622/real 1787828622] req@ffff8ab3d0a41e00 x1874671280784768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828622.182476:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280784768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828622.182485:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828622.182489:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828622.182492:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280784768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828622.182499:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280784896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828622.182519:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828622.182532:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828622/real 1787828622] req@ffff8ab3d0a430c0 x1874671280784896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828622.182544:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280784896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828622.182558:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828622.182562:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828622.182566:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280784896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828622.182571:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280785024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828622.182591:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828622.182602:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828622/real 1787828622] req@ffff8ab3d0a43c00 x1874671280785024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828622.182613:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280785024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828622.182622:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828622.182626:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828622.182629:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280785024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828622.182641:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280785152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828622.182660:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828622.182671:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828622/real 1787828622] req@ffff8ab3d0a43840 x1874671280785152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828622.182683:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280785152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828622.182693:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828622.182697:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828622.182700:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280785152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828623.206054:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828624.230098:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828625.254076:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828626.278013:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828627.302005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828627.302017:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828627.302022:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828627.302032:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3176 00000100:00080000:2.0:1787828627.302043:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1787828627.302045:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828627.302053:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828627.302091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280785280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828627.302113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828627.302116:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828627.302118:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828627.302120:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3176 00000100:00080000:2.0:1787828627.302123:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828627.302126:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828627.302135:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828627.302146:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280785408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828627.302151:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280785280:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828627.302167:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828627.302170:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828627.302171:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828627.302176:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3176 00000100:00080000:2.0:1787828627.302178:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828627.302180:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828627.302184:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828627.302193:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280785536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828627.302200:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828627.302203:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828627.302204:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828627.302222:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3176 00000100:00080000:2.0:1787828627.302227:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828627.302229:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828627.302234:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828627.302240:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828627.302250:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280785664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828627.302258:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828627.302260:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828627.302262:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828627.302265:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3176 00000100:00080000:2.0:1787828627.302268:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828627.302271:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828627.302275:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828627.302276:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828627/real 1787828627] req@ffff8ab3f0439a40 x1874671280785280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828627.302286:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280785792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828627.302295:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280785280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828627.302304:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828627.302308:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828627.302312:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280785280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828627.302330:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280785408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828627.302347:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828627.302355:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828627/real 1787828627] req@ffff8ab3f043a940 x1874671280785408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828627.302363:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280785408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828627.302369:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828627.302371:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828627.302373:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280785408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828627.302377:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280785536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828627.302386:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828627.302393:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828627/real 1787828627] req@ffff8ab3f043a1c0 x1874671280785536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828627.302400:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280785536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828627.302405:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828627.302407:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828627.302408:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280785536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828627.302412:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280785664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828627.302422:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828627.302428:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828627/real 1787828627] req@ffff8ab3f0439e00 x1874671280785664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828627.302435:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280785664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828627.302442:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828627.302443:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828627.302445:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280785664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828627.302448:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280785792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828627.302457:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828627.302463:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828627/real 1787828627] req@ffff8ab3f043b480 x1874671280785792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828627.302469:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280785792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828627.302476:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828627.302478:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828627.302479:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280785792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828628.326017:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828629.350028:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828630.374020:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828631.398025:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828632.421980:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828632.421989:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828632.421993:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828632.422001:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3182 00000100:00080000:2.0:1787828632.422010:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1787828632.422014:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828632.422019:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828632.422075:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280785920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828632.422098:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828632.422101:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828632.422103:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828632.422105:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3182 00000100:00080000:2.0:1787828632.422109:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828632.422113:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828632.422121:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828632.422129:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280785920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828632.422136:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280786048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828632.422151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828632.422153:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828632.422154:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828632.422157:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3182 00000100:00080000:2.0:1787828632.422160:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828632.422163:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828632.422167:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828632.422193:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280786176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828632.422201:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828632.422205:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828632.422206:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828632.422210:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828632.422214:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3182 00000100:00080000:2.0:1787828632.422218:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828632.422222:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828632.422227:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828632.422236:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280786304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828632.422243:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828632.422246:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828632.422248:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828632.422252:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3182 00000100:00100000:0.0:1787828632.422253:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828632/real 1787828632] req@ffff8ab3f043b840 x1874671280785920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828632.422257:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828632.422264:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828632.422272:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828632.422279:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280785920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828632.422283:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280786432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828632.422300:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828632.422305:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828632.422309:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280785920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828632.422329:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280786048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828632.422347:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828632.422356:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828632/real 1787828632] req@ffff8ab3f043b480 x1874671280786048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828632.422365:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280786048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828632.422371:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828632.422373:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828632.422375:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280786048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828632.422380:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280786176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828632.422390:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828632.422396:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828632/real 1787828632] req@ffff8ab3f0439e00 x1874671280786176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828632.422404:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280786176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828632.422410:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828632.422411:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828632.422413:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280786176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828632.422417:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280786304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828632.422427:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828632.422433:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828632/real 1787828632] req@ffff8ab3f043a1c0 x1874671280786304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828632.422441:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280786304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828632.422447:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828632.422450:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828632.422453:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280786304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828632.422459:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280786432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828632.422479:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828632.422492:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828632/real 1787828632] req@ffff8ab3f043a940 x1874671280786432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828632.422506:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280786432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828632.422518:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828632.422522:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828632.422526:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280786432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828633.446116:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828634.470052:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828635.494197:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828636.518096:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828637.542085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828637.542096:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828637.542101:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828637.542118:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3187 00010000:00010000:3.0:1787828637.542129:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828637.542131:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828637.542160:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828637.542221:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280786560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828637.542262:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828637.542265:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828637.542267:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828637.542270:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3187 00000100:00080000:0.0:1787828637.542273:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828637.542276:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828637.542287:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828637.542298:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280786688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828637.542310:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280786560:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828637.542325:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828637.542328:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828637.542330:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828637.542334:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3187 00000100:00080000:0.0:1787828637.542336:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828637.542339:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828637.542343:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828637.542352:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280786816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828637.542359:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828637.542362:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828637.542364:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828637.542366:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3187 00000100:00080000:0.0:1787828637.542384:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828637.542397:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828637.542400:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787828637.542408:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828637.542413:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280786944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828637.542425:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828637.542430:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828637.542432:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828637.542438:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3187 00000100:00080000:0.0:1787828637.542443:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828637.542448:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828637.542455:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828637.542470:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280787072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828637.542476:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828637/real 1787828637] req@ffff8ab3f075c000 x1874671280786560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828637.542503:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280786560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828637.542514:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828637.542517:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828637.542520:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280786560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828637.542549:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280786688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828637.542566:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828637.542586:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828637/real 1787828637] req@ffff8ab3f075cb40 x1874671280786688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828637.542595:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280786688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828637.542601:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828637.542603:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828637.542606:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280786688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828637.542610:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280786816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828637.542622:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828637.542627:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828637/real 1787828637] req@ffff8ab3f075f840 x1874671280786816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828637.542634:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280786816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828637.542639:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828637.542641:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828637.542643:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280786816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828637.542647:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280786944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828637.542658:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828637.542664:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828637/real 1787828637] req@ffff8ab3f075de00 x1874671280786944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828637.542671:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280786944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828637.542678:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828637.542680:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828637.542682:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280786944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828637.542685:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280787072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828637.542694:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828637.542701:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828637/real 1787828637] req@ffff8ab3f075f0c0 x1874671280787072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828637.542708:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280787072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828637.542715:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828637.542717:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828637.542719:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280787072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828638.566115:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828639.590103:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828640.614139:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828641.638054:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828642.662010:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828642.662022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828642.662027:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828642.662039:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3192 00000100:00080000:0.0:1787828642.662051:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828642.662059:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828642.662120:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280787200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828642.662142:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828642.662162:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828642.662167:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828642.662170:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828642.662174:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3192 00000100:00080000:0.0:1787828642.662180:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828642.662186:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828642.662200:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828642.662213:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280787200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828642.662222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280787328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828642.662239:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828642.662246:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828642.662249:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828642.662255:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3192 00000100:00080000:0.0:1787828642.662274:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828642.662282:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828642.662283:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828642.662293:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828642.662311:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280787456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828642.662323:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828642.662327:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828642.662330:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828642.662337:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3192 00000100:00080000:0.0:1787828642.662342:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828642.662344:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828642/real 1787828642] req@ffff8ab3f075cf00 x1874671280787200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828642.662354:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828642.662371:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828642.662376:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280787200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828642.662387:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828642.662394:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828642.662398:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280787200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828642.662402:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280787584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828642.662420:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828642.662421:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280787328:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787828642.662435:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828642.662439:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828642.662446:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3192 00000100:00080000:0.0:1787828642.662453:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828642.662457:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828642.662467:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828642.662477:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828642/real 1787828642] req@ffff8ab3f075e940 x1874671280787328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828642.662480:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828642.662501:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280787328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828642.662514:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280787712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828642.662529:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828642.662544:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828642.662547:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280787328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828642.662552:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280787456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828642.662577:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828642.662587:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828642/real 1787828642] req@ffff8ab3f075ed00 x1874671280787456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828642.662596:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280787456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828642.662608:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828642.662611:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828642.662613:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280787456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828642.662624:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280787584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828642.662636:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828642.662643:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828642/real 1787828642] req@ffff8ab3f075da40 x1874671280787584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828642.662651:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280787584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828642.662657:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828642.662659:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828642.662661:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280787584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828642.662665:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280787712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828642.662675:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828642.662681:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828642/real 1787828642] req@ffff8ab3f075f480 x1874671280787712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828642.662687:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280787712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828642.662693:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828642.662695:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828642.662697:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280787712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828643.686074:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828644.710133:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828645.734058:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828646.758058:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828647.782004:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828647.782012:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828647.782016:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828647.782024:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3197 00000100:00080000:0.0:1787828647.782033:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828647.782040:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828647.782089:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280787840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828647.782093:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828647.782113:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828647.782116:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828647.782117:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828647.782120:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3197 00000100:00080000:0.0:1787828647.782123:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828647.782126:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828647.782134:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828647.782145:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280787968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828647.782152:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828647.782155:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828647.782156:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828647.782158:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3197 00000100:00080000:0.0:1787828647.782162:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828647.782166:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828647.782169:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828647.782178:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280788096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828647.782189:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280787840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828647.782190:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828647.782199:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828647.782203:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828647.782208:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3197 00000100:00080000:0.0:1787828647.782210:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828647.782213:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828647.782215:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828647.782223:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280788224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828647.782229:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828647.782231:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828647.782232:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828647.782235:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3197 00000100:00080000:0.0:1787828647.782238:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828647.782240:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828647.782243:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828647.782250:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280788352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828647.782316:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828647.782399:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828647/real 1787828647] req@ffff8ab3f075e580 x1874671280787840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828647.782422:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280787840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828647.782435:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828647.782441:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828647.782446:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280787840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828647.782455:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280787968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828647.782483:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828647.782499:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828647/real 1787828647] req@ffff8ab3f075f480 x1874671280787968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828647.782514:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280787968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828647.782524:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828647.782527:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828647.782531:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280787968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828647.782565:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280788096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828647.782589:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828647.782612:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828647/real 1787828647] req@ffff8ab3f075da40 x1874671280788096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828647.782626:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280788096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828647.782634:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828647.782638:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828647.782641:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280788096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828647.782649:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280788224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828647.782671:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828647.782683:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828647/real 1787828647] req@ffff8ab3f075ed00 x1874671280788224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828647.782695:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280788224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828647.782707:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828647.782711:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828647.782715:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280788224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828647.782722:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280788352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828647.782743:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828647.782755:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828647/real 1787828647] req@ffff8ab3f075e940 x1874671280788352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828647.782768:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280788352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828647.782779:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828647.782783:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828647.782786:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280788352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828648.806022:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828649.830024:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828650.854081:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828651.878115:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828652.902086:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828652.902098:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828652.902103:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828652.902115:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3202 00010000:00010000:2.0:1787828652.902116:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828652.902126:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828652.902135:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828652.902192:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280788480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828652.902218:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828652.902221:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828652.902223:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828652.902226:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3202 00000100:00080000:3.0:1787828652.902229:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828652.902232:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828652.902242:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828652.902250:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280788480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828652.902254:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280788608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828652.902264:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828652.902266:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828652.902267:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828652.902270:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3202 00000100:00080000:3.0:1787828652.902272:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828652.902274:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828652.902277:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828652.902287:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280788736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828652.902299:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828652.902302:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828652.902304:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828652.902309:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3202 00000100:00080000:3.0:1787828652.902313:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828652.902319:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787828652.902324:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787828652.902326:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828652.902344:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280788864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828652.902353:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828652.902356:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828652.902358:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828652.902361:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3202 00000100:00080000:3.0:1787828652.902364:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828652.902367:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828652.902371:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828652.902381:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280788992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828652.902388:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828652/real 1787828652] req@ffff8ab3d0a40000 x1874671280788480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828652.902410:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280788480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828652.902419:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828652.902423:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828652.902427:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280788480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828652.902452:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280788608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828652.902470:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828652.902480:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828652/real 1787828652] req@ffff8ab3d0a421c0 x1874671280788608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828652.902488:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280788608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828652.902494:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828652.902497:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828652.902499:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280788608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828652.902503:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280788736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828652.902513:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828652.902519:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828652/real 1787828652] req@ffff8ab3d0a412c0 x1874671280788736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828652.902527:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280788736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828652.902533:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828652.902534:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828652.902536:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280788736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828652.902540:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280788864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828652.902551:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828652.902557:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828652/real 1787828652] req@ffff8ab3d0a43840 x1874671280788864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828652.902564:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280788864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828652.902571:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828652.902573:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828652.902575:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280788864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828652.902579:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280788992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828652.902589:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828652.902594:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828652/real 1787828652] req@ffff8ab3d0a43c00 x1874671280788992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828652.902602:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280788992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828652.902608:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828652.902610:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828652.902612:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280788992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828653.926127:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828654.950147:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828655.974148:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828656.998107:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828658.022060:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828658.022070:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828658.022076:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828658.022087:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3207 00000100:00080000:3.0:1787828658.022099:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828658.022109:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828658.022143:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828658.022158:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280789120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828658.022193:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828658.022197:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828658.022198:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828658.022201:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3207 00000100:00080000:3.0:1787828658.022203:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828658.022206:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828658.022217:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828658.022227:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280789248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828658.022237:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280789120:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828658.022237:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828658.022243:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828658.022247:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828658.022266:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3207 00000100:00080000:3.0:1787828658.022269:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828658.022273:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828658.022279:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828658.022293:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280789376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828658.022312:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828658.022315:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828658.022316:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828658.022320:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3207 00000100:00080000:3.0:1787828658.022324:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828658.022327:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828658.022332:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787828658.022340:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828658.022346:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280789504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828658.022354:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828658.022357:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828658.022358:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828658.022361:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3207 00000100:00080000:3.0:1787828658.022364:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828658.022366:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828658.022370:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828658.022378:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280789632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828658.022411:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828658/real 1787828658] req@ffff8ab3d0a430c0 x1874671280789120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828658.022444:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280789120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828658.022458:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828658.022463:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828658.022468:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280789120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828658.022519:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280789248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828658.022547:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828658.022563:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828658/real 1787828658] req@ffff8ab3d0a41e00 x1874671280789248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828658.022578:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280789248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828658.022588:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828658.022591:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828658.022595:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280789248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828658.022611:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280789376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828658.022635:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828658.022650:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828658/real 1787828658] req@ffff8ab3d0a43480 x1874671280789376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828658.022665:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280789376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828658.022683:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828658.022687:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828658.022691:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280789376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828658.022698:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280789504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828658.022720:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828658.022733:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828658/real 1787828658] req@ffff8ab3d0a43c00 x1874671280789504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828658.022759:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280789504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828658.022771:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828658.022774:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828658.022778:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280789504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828658.022785:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280789632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828658.022817:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828658.022831:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828658/real 1787828658] req@ffff8ab3d0a43840 x1874671280789632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828658.022845:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280789632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828658.022857:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828658.022861:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828658.022865:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280789632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828659.046096:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828660.070137:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828661.094108:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828662.118080:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828663.141972:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828663.141981:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828663.141984:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828663.142008:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3212 00000100:00080000:3.0:1787828663.142019:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828663.142026:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828663.142050:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828663.142062:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280789760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828663.142086:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828663.142089:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828663.142090:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828663.142093:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3212 00000100:00080000:3.0:1787828663.142096:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828663.142100:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828663.142109:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828663.142115:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280789760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828663.142138:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280789888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828663.142149:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828663.142152:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828663.142154:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828663.142158:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3212 00000100:00080000:3.0:1787828663.142161:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828663.142165:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828663.142169:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828663.142183:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280790016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828663.142196:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828663.142202:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828663.142205:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828663.142209:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828663.142216:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3212 00000100:00080000:3.0:1787828663.142221:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828663.142229:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828663.142236:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828663.142269:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280790144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828663.142276:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828663.142278:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828663/real 1787828663] req@ffff8ab3d0a412c0 x1874671280789760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828663.142279:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828663.142283:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828663.142291:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3212 00000100:00080000:3.0:1787828663.142295:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828663.142298:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828663.142303:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280789760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828663.142303:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:2.0:1787828663.142315:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828663.142316:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280790272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828663.142321:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828663.142330:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280789760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828663.142353:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280789888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828663.142369:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828663.142377:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828663/real 1787828663] req@ffff8ab3d0a421c0 x1874671280789888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828663.142385:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280789888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828663.142391:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828663.142393:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828663.142396:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280789888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828663.142400:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280790016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828663.142410:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828663.142415:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828663/real 1787828663] req@ffff8ab3d0a40000 x1874671280790016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828663.142422:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280790016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828663.142427:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828663.142429:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828663.142430:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280790016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828663.142434:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280790144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828663.142444:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828663.142449:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828663/real 1787828663] req@ffff8ab3d0a43840 x1874671280790144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828663.142456:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280790144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828663.142462:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828663.142464:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828663.142465:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280790144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828663.142469:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280790272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828663.142477:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828663.142483:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828663/real 1787828663] req@ffff8ab3d0a43c00 x1874671280790272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828663.142490:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280790272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828663.142496:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828663.142498:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828663.142499:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280790272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828664.166159:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828665.190072:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828666.214077:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828667.238095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828668.262011:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828668.262020:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828668.262024:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828668.262033:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3217 00000100:00080000:0.0:1787828668.262043:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828668.262049:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828668.262059:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828668.262083:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280790400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828668.262109:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828668.262114:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828668.262116:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828668.262119:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3217 00000100:00080000:0.0:1787828668.262123:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828668.262127:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828668.262135:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828668.262149:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280790528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828668.262149:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280790400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828668.262171:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828668.262173:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828668.262174:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828668.262178:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3217 00000100:00080000:0.0:1787828668.262180:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828668.262184:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828668.262189:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828668.262197:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280790656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828668.262204:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828668.262207:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828668.262208:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828668.262212:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3217 00000100:00080000:0.0:1787828668.262236:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828668.262239:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828668.262243:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828668.262251:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280790784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828668.262255:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828668.262266:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828668.262269:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828668.262271:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828668.262275:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3217 00000100:00080000:0.0:1787828668.262279:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828668.262283:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828668.262286:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828668.262296:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280790912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828668.262336:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828668/real 1787828668] req@ffff8ab3f075cf00 x1874671280790400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828668.262354:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280790400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828668.262364:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828668.262367:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828668.262371:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280790400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828668.262396:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280790528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828668.262416:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828668.262425:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828668/real 1787828668] req@ffff8ab3f075f0c0 x1874671280790528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828668.262434:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280790528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828668.262440:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828668.262442:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828668.262445:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280790528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828668.262449:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280790656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828668.262467:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828668.262479:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828668/real 1787828668] req@ffff8ab3f075de00 x1874671280790656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828668.262492:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280790656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828668.262510:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828668.262514:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828668.262516:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280790656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828668.262521:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280790784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828668.262539:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828668.262547:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828668/real 1787828668] req@ffff8ab3f075f840 x1874671280790784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828668.262555:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280790784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828668.262563:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828668.262565:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828668.262567:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280790784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828668.262572:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280790912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828668.262582:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828668.262588:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828668/real 1787828668] req@ffff8ab3f075cb40 x1874671280790912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828668.262595:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280790912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828668.262601:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828668.262603:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828668.262605:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280790912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828669.286040:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828670.310059:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828671.334087:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828672.358088:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828673.382063:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828673.382073:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828673.382076:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828673.382095:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3223 00000100:00080000:2.0:1787828673.382106:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828673.382134:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828673.382151:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828673.382173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280791040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828673.382211:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828673.382215:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828673.382218:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828673.382223:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3223 00000100:00080000:2.0:1787828673.382227:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828673.382236:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828673.382244:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828673.382253:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280791040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828673.382259:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280791168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828673.382273:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828673.382277:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828673.382279:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828673.382284:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3223 00000100:00080000:2.0:1787828673.382289:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828673.382293:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828673.382298:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828673.382312:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280791296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828673.382321:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828673.382325:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828673.382327:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828673.382338:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828673.382345:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3223 00000100:00080000:2.0:1787828673.382350:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828673.382355:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828673.382360:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828673.382376:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828673/real 1787828673] req@ffff8ab3f0439a40 x1874671280791040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828673.382378:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280791424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828673.382404:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828673.382405:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280791040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828673.382418:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828673.382422:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828673.382424:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828673.382428:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3223 00000100:00080000:0.0:1787828673.382431:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787828673.382434:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828673.382436:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280791040:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828673.382445:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828673.382452:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828673.382459:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280791168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828673.382470:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280791552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828673.382481:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828673.382498:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828673/real 1787828673] req@ffff8ab3f043ad00 x1874671280791168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828673.382514:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280791168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828673.382523:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828673.382527:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828673.382531:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280791168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828673.382537:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280791296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828673.382556:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828673.382564:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828673/real 1787828673] req@ffff8ab3f04392c0 x1874671280791296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828673.382573:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280791296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828673.382583:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828673.382585:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828673.382587:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280791296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828673.382591:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280791424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828673.382602:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828673.382608:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828673/real 1787828673] req@ffff8ab3f04383c0 x1874671280791424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828673.382615:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280791424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828673.382621:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828673.382623:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828673.382625:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280791424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828673.382635:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280791552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828673.382645:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828673.382651:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828673/real 1787828673] req@ffff8ab3f043bc00 x1874671280791552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828673.382658:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280791552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828673.382664:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828673.382665:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828673.382668:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280791552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828674.406108:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828675.430127:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828676.454076:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828677.478108:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828678.502002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828678.502012:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828678.502016:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828678.502026:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3228 00000100:00080000:2.0:1787828678.502036:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828678.502042:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828678.502079:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280791680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828678.502108:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828678.502123:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828678.502128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828678.502130:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828678.502133:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3228 00000100:00080000:2.0:1787828678.502135:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828678.502138:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828678.502158:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828678.502171:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280791808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828678.502182:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828678.502186:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828678.502188:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828678.502194:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3228 00000100:00080000:2.0:1787828678.502208:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828678.502215:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828678.502229:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280791680:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787828678.502232:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828678.502248:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280791936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828678.502258:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828678.502261:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828678.502263:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828678.502266:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3228 00000100:00080000:2.0:1787828678.502278:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828678.502282:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787828678.502286:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787828678.502290:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828678.502303:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280792064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828678.502310:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828678.502312:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828678.502314:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828678.502317:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3228 00000100:00080000:2.0:1787828678.502320:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828678.502323:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828678.502328:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828678.502335:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828678/real 1787828678] req@ffff8ab3f0439680 x1874671280791680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828678.502339:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280792192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828678.502366:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280791680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828678.502377:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828678.502381:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828678.502385:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280791680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828678.502391:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280791808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828678.502408:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828678.502416:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828678/real 1787828678] req@ffff8ab3f043a580 x1874671280791808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828678.502435:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280791808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828678.502441:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828678.502443:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828678.502446:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280791808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828678.502464:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280791936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828678.502477:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828678.502484:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828678/real 1787828678] req@ffff8ab3f0438b40 x1874671280791936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828678.502493:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280791936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828678.502498:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828678.502507:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828678.502510:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280791936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828678.502514:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280792064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828678.502526:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828678.502533:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828678/real 1787828678] req@ffff8ab3f0438780 x1874671280792064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828678.502541:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280792064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828678.502548:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828678.502550:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828678.502552:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280792064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828678.502556:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280792192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828678.502565:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828678.502574:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828678/real 1787828678] req@ffff8ab3f043b0c0 x1874671280792192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828678.502586:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280792192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828678.502598:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828678.502601:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828678.502605:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280792192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828679.526130:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828680.550115:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828681.574072:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828682.598056:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828683.622036:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1787828683.622050:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828683.622053:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828683.622062:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828683.622089:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3233 00000100:00080000:3.0:1787828683.622100:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828683.622107:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828683.622152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280792320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828683.622177:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828683.622179:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828683.622181:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828683.622183:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3233 00000100:00080000:3.0:1787828683.622186:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828683.622189:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828683.622198:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828683.622209:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280792448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828683.622221:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828683.622222:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280792320:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828683.622230:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828683.622233:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828683.622239:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3233 00000100:00080000:3.0:1787828683.622244:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828683.622249:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828683.622263:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828683.622277:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280792576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828683.622284:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828683.622287:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828683.622289:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828683.622293:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3233 00000100:00080000:3.0:1787828683.622306:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828683.622310:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828683.622315:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828683.622327:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280792704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828683.622338:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828683.622343:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828683.622346:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828683.622352:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3233 00000100:00080000:3.0:1787828683.622356:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828683.622362:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828683.622367:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828683.622382:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280792448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787828683.622394:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280792832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828683.622417:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280792576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828683.622435:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280792704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828683.622461:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280792832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828684.646174:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828685.670137:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787828686.694425:0:10942:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00010000:00010000:3.0:1787828686.694481:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787828686.694526:0:10942:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828686.694549:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787828686.694612:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787828686.694632:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787828686.694665:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787828686.694699:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:0.0:1787828686.694702:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828683/real 1787828686] req@ffff8ab3d0a41e00 x1874671280792448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828686.694746:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280792448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828686.694758:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828686.694765:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828686.694770:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280792448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828686.694780:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828683/real 1787828686] req@ffff8ab3d0a430c0 x1874671280792576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828686.694810:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280792576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828686.694831:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828686.694836:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828686.694839:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280792576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828686.694846:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828683/real 1787828686] req@ffff8ab3d0a43c00 x1874671280792704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828686.694860:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280792704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828686.694870:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828686.694874:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828686.694877:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280792704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828686.694904:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828683/real 1787828686] req@ffff8ab3d0a43480 x1874671280792320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828686.694918:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280792320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828686.694925:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828686.694928:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828686.694931:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280792320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828686.694962:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828683/real 1787828686] req@ffff8ab3d0a43840 x1874671280792832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828686.694975:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280792832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828686.694986:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828686.694991:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828686.694996:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280792832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828687.718118:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828688.741988:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828688.741999:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828688.742003:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828688.742014:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3238 00000100:00080000:2.0:1787828688.742036:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828688.742046:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828688.742114:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828688.742123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280792960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828688.742150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828688.742153:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828688.742155:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828688.742159:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3238 00000100:00080000:2.0:1787828688.742164:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828688.742169:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828688.742182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828688.742207:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280792960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828688.742213:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280793088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828688.742238:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828688.742243:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828688.742245:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828688.742251:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3238 00000100:00080000:2.0:1787828688.742255:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828688.742260:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828688.742276:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828688.742292:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280793216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828688.742303:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828688.742308:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787828688.742312:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828688.742312:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828688.742322:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3238 00000100:00080000:2.0:1787828688.742327:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828688.742333:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828688.742339:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828688.742352:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280793344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828688.742362:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828688.742365:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828688.742367:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828688.742373:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3238 00000100:00080000:2.0:1787828688.742377:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828688.742383:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828688/real 1787828688] req@ffff8ab3f0438f00 x1874671280792960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828688.742386:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828688.742396:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828688.742413:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280793472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828688.742417:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280792960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828688.742443:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828688.742449:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828688.742455:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280792960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828688.742488:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280793088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828688.742514:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828688.742529:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828688/real 1787828688] req@ffff8ab3f043b0c0 x1874671280793088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828688.742543:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280793088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828688.742552:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828688.742556:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828688.742559:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280793088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828688.742567:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280793216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828688.742589:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828688.742603:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828688/real 1787828688] req@ffff8ab3f0438780 x1874671280793216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828688.742616:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280793216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828688.742626:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828688.742629:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828688.742633:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280793216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828688.742640:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280793344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828688.742663:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828688.742683:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828688/real 1787828688] req@ffff8ab3f0438b40 x1874671280793344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828688.742696:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280793344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828688.742708:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828688.742711:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828688.742714:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280793344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828688.742721:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280793472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828688.742743:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828688.742758:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828688/real 1787828688] req@ffff8ab3f043a580 x1874671280793472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828688.742771:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280793472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828688.742800:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828688.742804:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828688.742808:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280793472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828689.766185:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828690.790095:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828691.814043:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828692.838071:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828693.861992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828693.862000:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828693.862003:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828693.862014:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3243 00000100:00080000:3.0:1787828693.862026:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828693.862036:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828693.862043:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828693.862076:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280793600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828693.862103:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828693.862106:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828693.862108:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828693.862110:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3243 00000100:00080000:3.0:1787828693.862113:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828693.862116:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828693.862124:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828693.862133:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280793728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828693.862140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828693.862143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828693.862144:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828693.862147:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3243 00000100:00080000:3.0:1787828693.862148:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828693.862151:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828693.862156:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280793600:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787828693.862157:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828693.862171:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280793856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828693.862179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828693.862181:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828693.862183:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828693.862185:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3243 00000100:00080000:3.0:1787828693.862187:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828693.862190:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828693.862194:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828693.862202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280793984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828693.862208:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828693.862211:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828693.862212:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828693.862215:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3243 00000100:00080000:3.0:1787828693.862217:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828693.862219:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828693.862222:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:0.0:1787828693.862224:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828693.862231:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280794112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828693.862283:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828693/real 1787828693] req@ffff8ab3d0a40000 x1874671280793600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828693.862304:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280793600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828693.862316:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828693.862319:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828693.862322:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280793600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828693.862328:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280793728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828693.862348:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828693.862357:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828693/real 1787828693] req@ffff8ab3d0a421c0 x1874671280793728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828693.862366:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280793728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828693.862376:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828693.862378:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828693.862380:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280793728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828693.862399:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280793856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828693.862409:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828693.862416:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828693/real 1787828693] req@ffff8ab3d0a412c0 x1874671280793856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828693.862423:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280793856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828693.862428:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828693.862430:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828693.862432:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280793856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828693.862435:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280793984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828693.862445:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828693.862451:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828693/real 1787828693] req@ffff8ab3d0a43840 x1874671280793984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828693.862458:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280793984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828693.862464:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828693.862466:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828693.862468:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280793984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828693.862472:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280794112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828693.862481:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828693.862490:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828693/real 1787828693] req@ffff8ab3d0a43480 x1874671280794112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828693.862502:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280794112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828693.862512:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828693.862515:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828693.862519:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280794112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828694.886070:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828695.910134:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828696.934053:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828697.958114:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828698.982102:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828698.982114:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828698.982119:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1787828698.982131:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828698.982133:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3248 00000100:00080000:0.0:1787828698.982148:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828698.982158:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828698.982206:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280794240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828698.982241:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828698.982246:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828698.982266:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828698.982280:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3248 00000100:00080000:0.0:1787828698.982284:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828698.982289:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828698.982301:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828698.982319:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280794368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828698.982332:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280794240:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828698.982337:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828698.982344:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828698.982346:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828698.982351:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3248 00000100:00080000:0.0:1787828698.982357:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828698.982360:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828698.982366:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828698.982378:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280794496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828698.982390:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828698.982394:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828698.982396:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828698.982399:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3248 00000100:00080000:0.0:1787828698.982407:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828698.982411:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828698.982417:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828698.982421:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828698.982467:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280794624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828698.982480:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828698.982485:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828698.982488:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828698.982494:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3248 00000100:00080000:0.0:1787828698.982500:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828698.982505:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828698.982513:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828698.982526:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828698/real 1787828698] req@ffff8ab3f075c000 x1874671280794240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828698.982534:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280794752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828698.982594:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280794240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828698.982611:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828698.982617:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828698.982622:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280794240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828698.982652:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280794368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828698.982682:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828698.982698:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828698/real 1787828698] req@ffff8ab3f075cb40 x1874671280794368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828698.982757:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280794368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828698.982767:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828698.982771:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828698.982774:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280794368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828698.982782:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280794496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828698.982825:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828698.982839:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828698/real 1787828698] req@ffff8ab3f075f840 x1874671280794496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828698.982854:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280794496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828698.982864:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828698.982867:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828698.982871:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280794496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828698.982878:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280794624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828698.982898:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828698.982975:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828698/real 1787828698] req@ffff8ab3f075de00 x1874671280794624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828698.982998:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280794624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828698.983016:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828698.983020:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828698.983024:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280794624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828698.983032:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280794752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828698.983067:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828698.983080:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828698/real 1787828698] req@ffff8ab3f075f0c0 x1874671280794752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828698.983102:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280794752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828698.983113:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828698.983116:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828698.983119:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280794752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828700.006162:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828701.030056:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828702.054064:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828703.077995:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828704.102074:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828704.102088:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828704.102094:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828704.102108:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3253 00010000:00010000:2.0:1787828704.102118:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828704.102121:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828704.102135:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828704.102184:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280794880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828704.102242:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828704.102250:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828704.102254:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828704.102263:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3253 00000100:00080000:3.0:1787828704.102272:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828704.102282:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828704.102299:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828704.102323:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280794880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828704.102333:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280795008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828704.102352:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828704.102356:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828704.102359:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828704.102364:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3253 00000100:00080000:3.0:1787828704.102370:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828704.102377:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828704.102385:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828704.102403:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280795136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828704.102416:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828704.102427:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787828704.102428:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828704.102430:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828704.102440:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3253 00000100:00080000:3.0:1787828704.102447:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828704.102453:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828704.102460:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828704.102490:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280795264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828704.102504:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828704.102509:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828704.102511:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828704.102517:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3253 00000100:00100000:1.0:1787828704.102525:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828704/real 1787828704] req@ffff8ab3d0a43c00 x1874671280794880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828704.102526:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828704.102538:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828704.102544:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828704.102567:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280795392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828704.102577:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280794880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828704.102601:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828704.102607:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828704.102615:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280794880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828704.102647:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280795008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828704.102714:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828704.102736:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828704/real 1787828704] req@ffff8ab3d0a430c0 x1874671280795008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828704.102753:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280795008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828704.102765:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828704.102770:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828704.102777:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280795008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828704.102790:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280795136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828704.102830:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828704.102854:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828704/real 1787828704] req@ffff8ab3d0a41e00 x1874671280795136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828704.102879:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280795136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828704.102922:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828704.102927:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828704.102931:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280795136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828704.102940:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280795264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828704.102971:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828704.102987:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828704/real 1787828704] req@ffff8ab3d0a43480 x1874671280795264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828704.103002:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280795264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828704.103013:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828704.103017:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828704.103022:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280795264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828704.103029:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280795392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828704.103048:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828704.103058:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828704/real 1787828704] req@ffff8ab3d0a43840 x1874671280795392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828704.103071:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280795392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828704.103088:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828704.103095:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828704.103102:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280795392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787828705.126052:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828706.150018:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828707.174034:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828708.198037:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828709.221991:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828709.222002:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828709.222006:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828709.222017:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3258 00000100:00080000:3.0:1787828709.222026:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1787828709.222027:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828709.222035:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828709.222089:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280795520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828709.222130:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828709.222133:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828709.222134:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828709.222137:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3258 00000100:00080000:3.0:1787828709.222139:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828709.222142:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828709.222149:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828709.222160:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280795648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828709.222162:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280795520:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828709.222176:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828709.222197:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828709.222199:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828709.222204:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3258 00000100:00080000:3.0:1787828709.222207:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828709.222210:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828709.222215:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828709.222227:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280795776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828709.222241:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828709.222243:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828709.222245:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828709.222252:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828709.222252:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3258 00000100:00080000:3.0:1787828709.222261:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828709.222265:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828709.222269:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828709.222280:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280795904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828709.222288:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828709.222291:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828709.222293:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828709.222296:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3258 00000100:00080000:3.0:1787828709.222299:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828709.222303:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828709.222308:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828709.222317:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828709/real 1787828709] req@ffff8ab3d0a412c0 x1874671280795520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828709.222320:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280796032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828709.222394:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280795520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828709.222436:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828709.222441:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828709.222445:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280795520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828709.222472:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280795648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828709.222495:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828709.222524:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828709/real 1787828709] req@ffff8ab3d0a421c0 x1874671280795648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828709.222533:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280795648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828709.222539:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828709.222541:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828709.222544:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280795648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828709.222547:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280795776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828709.222559:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828709.222564:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828709/real 1787828709] req@ffff8ab3d0a40000 x1874671280795776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828709.222588:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280795776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828709.222596:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828709.222598:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828709.222601:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280795776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828709.222606:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280795904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828709.222620:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828709.222628:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828709/real 1787828709] req@ffff8ab3d0a43840 x1874671280795904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828709.222656:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280795904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828709.222689:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828709.222694:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828709.222700:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280795904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828709.222709:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280796032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828709.222739:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828709.222760:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828709/real 1787828709] req@ffff8ab3d0a43480 x1874671280796032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828709.222776:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280796032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828709.222790:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828709.222795:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828709.222800:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280796032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828710.246068:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828711.270023:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828712.294070:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828713.317969:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828714.341968:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828714.341980:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828714.341984:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828714.341994:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3263 00000100:00080000:1.0:1787828714.342005:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828714.342014:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828714.342060:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280796160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828714.342083:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828714.342086:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828714.342087:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828714.342090:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3263 00000100:00080000:1.0:1787828714.342093:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828714.342096:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828714.342103:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828714.342113:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280796288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828714.342121:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828714.342124:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828714.342125:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828714.342128:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3263 00000100:00080000:1.0:1787828714.342129:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828714.342132:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828714.342136:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828714.342145:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280796160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828714.342145:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280796416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828714.342160:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828714.342162:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828714.342164:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828714.342167:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3263 00000100:00080000:1.0:1787828714.342168:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828714.342171:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828714.342174:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828714.342181:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280796544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828714.342188:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828714.342190:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828714.342191:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828714.342194:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3263 00000100:00080000:1.0:1787828714.342203:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828714.342205:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787828714.342213:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:1.0:1787828714.342214:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828714.342224:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280796672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828714.342259:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828714/real 1787828714] req@ffff8ab3d22fd680 x1874671280796160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828714.342275:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280796160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828714.342284:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828714.342287:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828714.342291:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280796160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828714.342298:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280796288:192.168.203.124@tcp:38:kworker.0 00010000:00010000:2.0:1787828714.342309:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:0.0:1787828714.342315:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828714.342326:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828714/real 1787828714] req@ffff8ab3d22fcb40 x1874671280796288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828714.342334:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280796288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828714.342340:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828714.342342:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828714.342345:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280796288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828714.342362:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280796416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828714.342372:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828714.342378:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828714/real 1787828714] req@ffff8ab3d22fda40 x1874671280796416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828714.342385:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280796416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828714.342390:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828714.342392:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828714.342394:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280796416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828714.342397:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280796544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828714.342407:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828714.342413:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828714/real 1787828714] req@ffff8ab3d22fc000 x1874671280796544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828714.342420:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280796544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828714.342426:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828714.342427:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828714.342430:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280796544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828714.342433:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280796672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828714.342443:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828714.342448:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828714/real 1787828714] req@ffff8ab3d22ff480 x1874671280796672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828714.342454:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280796672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828714.342461:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828714.342462:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828714.342464:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280796672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787828715.366023:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828716.390065:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828717.414039:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828718.438082:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828719.462025:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828719.462034:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828719.462038:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828719.462047:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3269 00010000:00010000:1.0:1787828719.462054:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828719.462056:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828719.462063:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828719.462097:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280796800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828719.462125:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828719.462130:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828719.462132:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828719.462136:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3269 00000100:00080000:2.0:1787828719.462140:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828719.462144:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828719.462152:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828719.462165:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280796928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828719.462175:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828719.462177:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828719.462179:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828719.462184:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280796800:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828719.462185:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3269 00000100:00080000:2.0:1787828719.462190:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828719.462195:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828719.462198:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828719.462208:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280797056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828719.462214:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828719.462217:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828719.462218:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828719.462221:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3269 00000100:00080000:2.0:1787828719.462239:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828719.462242:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828719.462245:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787828719.462251:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828719.462256:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280797184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828719.462265:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828719.462268:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828719.462269:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828719.462272:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3269 00000100:00080000:2.0:1787828719.462275:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828719.462277:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828719.462280:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828719.462289:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280797312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828719.462303:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828719/real 1787828719] req@ffff8ab3f0439680 x1874671280796800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828719.462324:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280796800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828719.462349:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828719.462353:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828719.462358:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280796800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828719.462380:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280796928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828719.462402:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828719.462426:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828719/real 1787828719] req@ffff8ab3f043bc00 x1874671280796928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828719.462439:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280796928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828719.462448:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828719.462451:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828719.462455:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280796928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828719.462461:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280797056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828719.462488:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828719.462499:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828719/real 1787828719] req@ffff8ab3f04383c0 x1874671280797056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828719.462511:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280797056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828719.462520:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828719.462523:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828719.462527:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280797056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828719.462533:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280797184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828719.462549:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828719.462559:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828719/real 1787828719] req@ffff8ab3f04392c0 x1874671280797184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828719.462596:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280797184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828719.462608:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828719.462611:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828719.462614:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280797184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828719.462621:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280797312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828719.462640:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828719.462661:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828719/real 1787828719] req@ffff8ab3f043ad00 x1874671280797312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828719.462672:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280797312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828719.462682:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828719.462685:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828719.462688:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280797312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828720.486074:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828721.510081:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828722.534082:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828723.558044:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828724.582009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828724.582019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828724.582023:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828724.582032:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3274 00000100:00080000:2.0:1787828724.582041:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828724.582047:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828724.582061:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828724.582081:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280797440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828724.582104:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828724.582106:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828724.582108:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828724.582110:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3274 00000100:00080000:2.0:1787828724.582112:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828724.582115:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828724.582122:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828724.582131:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280797568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828724.582139:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828724.582145:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280797440:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828724.582145:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828724.582148:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828724.582156:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3274 00000100:00080000:2.0:1787828724.582159:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828724.582161:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828724.582164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828724.582173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280797696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828724.582180:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828724.582182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828724.582183:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828724.582186:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3274 00000100:00080000:2.0:1787828724.582200:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828724.582203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828724.582206:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787828724.582212:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828724.582216:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280797824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828724.582223:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828724.582226:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828724.582228:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828724.582231:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3274 00000100:00080000:2.0:1787828724.582235:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828724.582237:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828724.582241:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828724.582249:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280797952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828724.582264:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828724/real 1787828724] req@ffff8ab3f0439a40 x1874671280797440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828724.582279:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280797440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828724.582287:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828724.582290:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828724.582294:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280797440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828724.582335:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280797568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828724.582360:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828724.582374:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828724/real 1787828724] req@ffff8ab3f043a940 x1874671280797568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828724.582385:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280797568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828724.582391:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828724.582394:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828724.582396:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280797568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828724.582400:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280797696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828724.582414:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828724.582421:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828724/real 1787828724] req@ffff8ab3f043a1c0 x1874671280797696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828724.582428:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280797696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828724.582434:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828724.582436:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828724.582438:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280797696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828724.582441:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280797824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828724.582451:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828724.582466:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828724/real 1787828724] req@ffff8ab3f0439e00 x1874671280797824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828724.582473:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280797824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828724.582480:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828724.582481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828724.582483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280797824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828724.582487:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280797952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828724.582495:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828724.582501:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828724/real 1787828724] req@ffff8ab3f043b480 x1874671280797952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828724.582508:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280797952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828724.582514:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828724.582515:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828724.582517:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280797952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787828725.606049:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828726.630035:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828727.653978:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828728.678063:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828729.702028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828729.702038:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828729.702042:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828729.702050:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3279 00000100:00080000:2.0:1787828729.702060:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828729.702068:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:1.0:1787828729.702069:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828729.702105:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280798080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828729.702133:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828729.702136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828729.702137:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828729.702140:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3279 00000100:00080000:2.0:1787828729.702142:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828729.702144:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828729.702151:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828729.702161:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280798208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828729.702169:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828729.702172:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828729.702173:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828729.702176:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3279 00000100:00080000:2.0:1787828729.702179:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828729.702182:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828729.702186:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828729.702194:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280798336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828729.702204:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828729.702211:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280798080:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828729.702211:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828729.702217:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828729.702228:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3279 00000100:00080000:2.0:1787828729.702231:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828729.702234:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828729.702237:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828729.702248:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280798464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828729.702255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828729.702258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828729.702260:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828729.702263:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3279 00000100:00080000:2.0:1787828729.702265:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828729.702269:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828729.702272:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828729.702280:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280798592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828729.702304:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828729.702369:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828729/real 1787828729] req@ffff8ab3f043b840 x1874671280798080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828729.702384:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280798080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828729.702395:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828729.702398:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828729.702402:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280798080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828729.702408:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280798208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828729.702422:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828729.702430:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828729/real 1787828729] req@ffff8ab3f043b480 x1874671280798208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828729.702438:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280798208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828729.702443:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828729.702445:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828729.702447:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280798208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828729.702472:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280798336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828729.702483:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828729.702489:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828729/real 1787828729] req@ffff8ab3f0439e00 x1874671280798336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828729.702513:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280798336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828729.702530:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828729.702533:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828729.702538:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280798336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828729.702545:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280798464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828729.702564:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828729.702571:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828729/real 1787828729] req@ffff8ab3f043a1c0 x1874671280798464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828729.702579:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280798464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828729.702586:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828729.702587:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828729.702589:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280798464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828729.702593:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280798592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828729.702611:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828729.702624:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828729/real 1787828729] req@ffff8ab3f043a940 x1874671280798592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828729.702632:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280798592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828729.702638:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828729.702640:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828729.702642:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280798592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828730.726007:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828731.750062:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828732.774094:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828733.798049:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828734.821975:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828734.821983:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828734.821987:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828734.821997:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3284 00000100:00080000:2.0:1787828734.822005:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1787828734.822007:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828734.822012:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828734.822046:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280798720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828734.822067:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828734.822070:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828734.822071:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828734.822073:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3284 00000100:00080000:2.0:1787828734.822076:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828734.822078:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828734.822086:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828734.822103:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280798848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828734.822108:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280798720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828734.822122:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828734.822125:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828734.822126:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828734.822130:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3284 00000100:00080000:2.0:1787828734.822132:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828734.822134:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828734.822139:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828734.822148:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280798976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828734.822156:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828734.822158:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828734.822160:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828734.822163:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3284 00000100:00080000:2.0:1787828734.822165:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828734.822168:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828734.822172:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787828734.822175:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828734.822183:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280799104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828734.822191:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828734.822194:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828734.822195:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828734.822198:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3284 00000100:00080000:2.0:1787828734.822200:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828734.822203:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828734.822206:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828734.822215:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280799232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828734.822222:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828734/real 1787828734] req@ffff8ab3f0439a40 x1874671280798720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828734.822238:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280798720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828734.822247:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828734.822251:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828734.822256:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280798720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828734.822279:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280798848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828734.822304:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828734.822315:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828734/real 1787828734] req@ffff8ab3f043ad00 x1874671280798848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828734.822335:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280798848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828734.822341:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828734.822343:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828734.822345:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280798848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828734.822349:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280798976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828734.822362:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828734.822368:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828734/real 1787828734] req@ffff8ab3f04392c0 x1874671280798976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828734.822375:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280798976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828734.822396:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828734.822398:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828734.822400:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280798976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828734.822404:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280799104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828734.822414:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828734.822420:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828734/real 1787828734] req@ffff8ab3f04383c0 x1874671280799104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828734.822432:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280799104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828734.822442:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828734.822445:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828734.822449:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280799104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828734.822455:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280799232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828734.822489:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828734.822497:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828734/real 1787828734] req@ffff8ab3f043bc00 x1874671280799232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828734.822505:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280799232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828734.822512:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828734.822514:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828734.822517:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280799232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787828735.846016:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828736.870107:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828737.894107:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828738.918050:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828739.942084:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828739.942126:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828739.942139:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828739.942146:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828739.942156:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3289 00000100:00080000:0.0:1787828739.942167:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828739.942175:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828739.942219:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280799360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828739.942254:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828739.942258:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828739.942260:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828739.942264:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3289 00000100:00080000:0.0:1787828739.942269:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828739.942274:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828739.942285:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828739.942320:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280799488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828739.942334:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280799360:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828739.942340:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828739.942353:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828739.942364:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828739.942370:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3289 00000100:00080000:0.0:1787828739.942375:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828739.942381:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828739.942388:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828739.942405:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280799616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828739.942416:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828739.942420:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828739.942447:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828739.942458:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3289 00000100:00080000:0.0:1787828739.942492:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828739.942498:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828739.942505:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828739.942507:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828739.942528:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280799744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828739.942570:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828739.942576:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828739.942581:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828739.942592:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3289 00000100:00080000:0.0:1787828739.942600:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828739.942607:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828739/real 1787828739] req@ffff8ab3f075cf00 x1874671280799360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828739.942618:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828739.942631:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828739.942686:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280799872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828739.942690:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280799360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828739.942746:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828739.942752:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828739.942757:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280799360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828739.942787:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280799488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828739.942826:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828739.942841:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828739/real 1787828739] req@ffff8ab3f075e940 x1874671280799488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828739.942855:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280799488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828739.942864:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828739.942868:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828739.942871:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280799488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828739.942903:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280799616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828739.942971:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828739.942987:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828739/real 1787828739] req@ffff8ab3f075ed00 x1874671280799616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828739.943002:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280799616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828739.943023:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828739.943027:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828739.943031:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280799616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828739.943038:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280799744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828739.943077:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828739.943088:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828739/real 1787828739] req@ffff8ab3f075da40 x1874671280799744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828739.943100:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280799744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828739.943110:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828739.943113:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828739.943116:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280799744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828739.943122:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280799872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828739.943140:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828739.943157:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828739/real 1787828739] req@ffff8ab3f075f480 x1874671280799872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828739.943176:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280799872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828739.943186:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828739.943189:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828739.943192:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280799872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828740.966060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828741.989984:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828743.014136:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828744.038052:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828745.061984:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828745.061995:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828745.062000:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828745.062012:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3294 00010000:00010000:3.0:1787828745.062021:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828745.062023:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828745.062034:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828745.062071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280800000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828745.062095:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828745.062098:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828745.062100:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828745.062103:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3294 00000100:00080000:2.0:1787828745.062106:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828745.062110:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828745.062119:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828745.062130:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280800000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828745.062131:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280800128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828745.062146:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828745.062149:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828745.062150:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828745.062153:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3294 00000100:00080000:2.0:1787828745.062155:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828745.062158:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828745.062161:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828745.062173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280800256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828745.062186:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828745.062189:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828745.062190:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828745.062195:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3294 00000800:00000100:1.0:1787828745.062196:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828745.062199:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828745.062204:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828745.062209:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828745.062219:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280800384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828745.062226:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828745.062229:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828745.062231:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828745.062235:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3294 00000100:00080000:2.0:1787828745.062238:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828745.062241:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828745.062263:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828745/real 1787828745] req@ffff8ab3f0439680 x1874671280800000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828745.062264:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828745.062284:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280800512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828745.062295:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280800000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828745.062310:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828745.062316:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828745.062321:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280800000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828745.062346:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280800128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828745.062370:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828745.062382:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828745/real 1787828745] req@ffff8ab3f043a580 x1874671280800128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828745.062437:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280800128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828745.062444:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828745.062447:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828745.062450:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280800128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828745.062454:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280800256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828745.062470:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828745.062477:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828745/real 1787828745] req@ffff8ab3f0438b40 x1874671280800256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828745.062507:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280800256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828745.062516:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828745.062520:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828745.062524:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280800256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828745.062530:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280800384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828745.062549:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828745.062559:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828745/real 1787828745] req@ffff8ab3f0438780 x1874671280800384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828745.062572:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280800384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828745.062582:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828745.062586:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828745.062593:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280800384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828745.062619:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280800512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828745.062646:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828745.062664:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828745/real 1787828745] req@ffff8ab3f043b0c0 x1874671280800512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828745.062702:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280800512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828745.062712:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828745.062715:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828745.062718:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280800512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787828746.086058:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828747.110135:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828748.133989:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828749.158056:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828750.181957:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828750.181971:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828750.181978:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828750.181991:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3299 00000100:00080000:2.0:1787828750.182004:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1787828750.182018:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828750.182027:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828750.182073:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280800640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828750.182110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828750.182114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828750.182116:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828750.182119:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3299 00000100:00080000:2.0:1787828750.182124:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828750.182129:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828750.182150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828750.182163:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280800768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828750.182174:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828750.182180:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828750.182182:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280800640:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828750.182183:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828750.182193:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3299 00000100:00080000:2.0:1787828750.182198:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828750.182201:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828750.182216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828750.182230:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280800896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828750.182244:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828750.182249:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828750.182252:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828750.182259:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3299 00000100:00080000:2.0:1787828750.182275:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828750.182283:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828750.182298:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828750.182314:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280801024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828750.182324:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828750.182327:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828750.182329:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828750.182334:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3299 00000100:00080000:2.0:1787828750.182339:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828750.182343:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828750.182349:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828750.182381:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280801152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828750.182401:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280800768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828750.182455:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280800896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828750.182495:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280801024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828750.182534:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280801152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828751.206067:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828752.229999:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828753.254100:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828753.254200:0:10944:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787828753.254240:0:10944:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828753.254257:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828753.254292:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828753.254301:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828753.254314:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828753.254326:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787828753.254356:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828750/real 1787828753] req@ffff8ab3f043b0c0 x1874671280800768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828753.254412:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280800768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.254444:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828753.254452:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828753.254458:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280800768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828753.254468:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828750/real 1787828753] req@ffff8ab3f0438780 x1874671280800896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828753.254488:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280800896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.254502:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828753.254508:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828753.254513:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280800896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828753.254523:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828750/real 1787828753] req@ffff8ab3f0438b40 x1874671280801024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828753.254543:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280801024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.254558:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828753.254563:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828753.254569:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280801024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828753.254578:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828750/real 1787828753] req@ffff8ab3f0438f00 x1874671280800640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828753.254594:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280800640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.254606:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828753.254611:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828753.254615:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280800640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828753.254646:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828750/real 1787828753] req@ffff8ab3f043a580 x1874671280801152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828753.254663:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280801152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.254676:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828753.254681:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828753.254686:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280801152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787828754.278086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828755.302037:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828755.302051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828755.302056:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828755.302071:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3304 00000100:00080000:2.0:1787828755.302097:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828755.302109:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828755.302172:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828755.302173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280801280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828755.302219:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828755.302224:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828755.302226:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828755.302231:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3304 00000100:00080000:2.0:1787828755.302235:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828755.302241:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828755.302255:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828755.302278:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280801408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828755.302293:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828755.302301:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828755.302304:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280801280:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828755.302341:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828755.302348:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3304 00000100:00080000:2.0:1787828755.302353:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828755.302358:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828755.302364:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828755.302378:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280801536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828755.302390:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828755.302394:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828755.302396:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828755.302401:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3304 00000100:00080000:2.0:1787828755.302405:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828755.302410:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828755.302415:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828755.302425:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280801664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828755.302436:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828755.302440:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828755.302442:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828755.303115:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828755.303249:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828755/real 1787828755] req@ffff8ab3f0439680 x1874671280801280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828755.303289:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280801280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828755.303310:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828755.303346:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828755.303357:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280801280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828755.303407:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280801408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828755.303466:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828755.303509:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828755/real 1787828755] req@ffff8ab3f043bc00 x1874671280801408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828755.303537:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280801408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828755.303555:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828755.303561:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828755.303568:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280801408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828755.303581:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280801536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828755.303627:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828755.303650:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828755/real 1787828755] req@ffff8ab3f04383c0 x1874671280801536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828755.304754:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3304 00000100:00100000:1.0:1787828755.304756:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280801536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828755.304767:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828755.304783:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787828755.304784:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828755.304799:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000008:00100000:2.0:1787828755.304822:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828755.304824:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280801536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828755.304841:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280801664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828755.304856:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280801792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828755.304889:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828755.304941:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828755/real 1787828755] req@ffff8ab3f04392c0 x1874671280801664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828755.304957:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280801664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828755.304969:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828755.304973:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828755.304977:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280801664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828755.305011:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280801792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828755.305033:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828755.305045:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828755/real 1787828755] req@ffff8ab3f043ad00 x1874671280801792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828755.305058:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280801792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828755.305083:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828755.305087:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828755.305091:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280801792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787828756.326056:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828757.350075:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828758.374057:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828759.398088:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828760.422065:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828760.422065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828760.422079:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828760.422086:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828760.422101:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3310 00000100:00080000:3.0:1787828760.422133:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828760.422144:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828760.422215:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280801920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828760.422258:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828760.422264:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828760.422268:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828760.422275:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3310 00000100:00080000:3.0:1787828760.422307:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828760.422319:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828760.422320:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280801920:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787828760.422344:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828760.422368:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280802048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828760.422386:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828760.422391:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828760.422394:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828760.422400:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3310 00000100:00080000:3.0:1787828760.422406:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828760.422413:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828760.422421:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828760.422427:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828760.422444:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280802176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828760.422461:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828760.422466:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828760.422469:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828760.422474:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3310 00000100:00080000:3.0:1787828760.422481:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828760.422488:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828760.422498:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828760.422512:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828760/real 1787828760] req@ffff8ab3d0a41e00 x1874671280801920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828760.422522:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280802304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828760.422548:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828760.422555:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828760.422559:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828760.422563:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280801920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828760.422572:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3310 00000100:00080000:3.0:1787828760.422591:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828760.422594:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828760.422618:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787828760.422621:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828760.422629:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280801920:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787828760.422632:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828760.422652:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280802432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828760.422660:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280802048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828760.422703:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828760.422715:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828760/real 1787828760] req@ffff8ab3d0a430c0 x1874671280802048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828760.422726:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280802048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828760.422741:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828760.422744:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828760.422746:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280802048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828760.422752:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280802176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828760.422766:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828760.422774:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828760/real 1787828760] req@ffff8ab3d0a43c00 x1874671280802176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828760.422783:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280802176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828760.422798:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828760.422802:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828760.422806:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280802176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828760.422813:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280802304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828760.422837:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828760.422861:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828760/real 1787828760] req@ffff8ab3d0a43480 x1874671280802304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828760.422874:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280802304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828760.422883:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828760.422905:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828760.422908:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280802304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828760.422923:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280802432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828760.422944:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828760.422958:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828760/real 1787828760] req@ffff8ab3d0a43840 x1874671280802432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828760.422973:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280802432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828760.422986:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828760.422990:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828760.422995:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280802432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828761.446109:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828762.469995:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828763.494120:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828764.518000:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828765.542018:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828765.542028:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828765.542033:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828765.542055:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3315 00000100:00080000:3.0:1787828765.542067:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828765.542090:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828765.542168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280802560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828765.542184:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828765.542204:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828765.542210:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828765.542213:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828765.542218:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3315 00000100:00080000:3.0:1787828765.542224:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828765.542229:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828765.542242:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828765.542304:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280802560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828765.542310:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280802688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828765.542325:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828765.542328:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828765.542330:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828765.542334:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3315 00000100:00080000:3.0:1787828765.542338:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828765.542342:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828765.542347:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828765.542358:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280802816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828765.542366:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828765.542369:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828765.542399:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828765.542402:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3315 00000100:00080000:3.0:1787828765.542405:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828765.542409:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828765.542414:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828765.542416:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828765.542430:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280802944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828765.542441:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828765.542444:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828765.542446:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828765.542451:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3315 00000100:00080000:3.0:1787828765.542454:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828765.542458:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828765.542463:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828765.542475:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280803072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828765.542485:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828765/real 1787828765] req@ffff8ab3d0a40000 x1874671280802560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828765.542509:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280802560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828765.542520:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828765.542524:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828765.542528:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280802560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828765.542552:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280802688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828765.542581:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828765.542593:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828765/real 1787828765] req@ffff8ab3d0a421c0 x1874671280802688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828765.542611:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280802688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828765.542639:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828765.542643:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828765.542648:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280802688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828765.542657:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280802816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828765.542684:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828765.542698:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828765/real 1787828765] req@ffff8ab3d0a412c0 x1874671280802816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828765.542709:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280802816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828765.542716:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828765.542719:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828765.542721:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280802816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828765.542726:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280802944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828765.542743:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828765.542750:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828765/real 1787828765] req@ffff8ab3d0a43840 x1874671280802944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828765.542760:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280802944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828765.542767:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828765.542770:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828765.542772:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280802944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828765.542777:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280803072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828765.542788:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828765.542795:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828765/real 1787828765] req@ffff8ab3d0a43480 x1874671280803072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828765.542810:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280803072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828765.542818:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828765.542820:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828765.542823:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280803072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787828766.566016:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828767.590074:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828768.614039:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828769.638065:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828770.661982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828770.661991:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828770.661995:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828770.662003:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3320 00000100:00080000:3.0:1787828770.662011:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1787828770.662017:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828770.662017:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828770.662052:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280803200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828770.662073:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828770.662075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828770.662077:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828770.662079:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3320 00000100:00080000:3.0:1787828770.662082:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828770.662084:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828770.662091:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828770.662100:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280803328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828770.662109:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280803200:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828770.662110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828770.662116:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828770.662118:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828770.662121:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3320 00000100:00080000:3.0:1787828770.662123:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828770.662126:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828770.662129:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828770.662138:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280803456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828770.662145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828770.662148:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828770.662149:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828770.662152:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3320 00000100:00080000:3.0:1787828770.662156:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828770.662159:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828770.662162:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828770.662167:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828770.662173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280803584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828770.662181:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828770.662184:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828770.662185:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828770.662187:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3320 00000100:00080000:3.0:1787828770.662189:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828770.662192:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828770.662196:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828770.662203:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280803712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828770.662227:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828770/real 1787828770] req@ffff8ab3d0a43c00 x1874671280803200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828770.662246:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280803200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828770.662254:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828770.662257:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828770.662260:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280803200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828770.662278:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280803328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828770.662295:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828770.662304:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828770/real 1787828770] req@ffff8ab3d0a430c0 x1874671280803328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828770.662317:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280803328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828770.662323:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828770.662325:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828770.662327:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280803328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828770.662331:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280803456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828770.662340:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828770.662346:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828770/real 1787828770] req@ffff8ab3d0a41e00 x1874671280803456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828770.662354:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280803456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828770.662359:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828770.662360:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828770.662362:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280803456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828770.662366:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280803584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828770.662375:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828770.662381:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828770/real 1787828770] req@ffff8ab3d0a43480 x1874671280803584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828770.662388:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280803584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828770.662398:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828770.662400:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828770.662403:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280803584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828770.662406:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280803712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828770.662415:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828770.662420:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828770/real 1787828770] req@ffff8ab3d0a43840 x1874671280803712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828770.662435:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280803712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828770.662446:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828770.662449:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828770.662453:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280803712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828771.686017:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828772.710003:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828773.734028:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828774.758024:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828775.782084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828775.782094:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828775.782099:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828775.782110:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3325 00000100:00080000:3.0:1787828775.782120:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828775.782128:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828775.782159:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828775.782167:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280803840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828775.782241:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828775.782245:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828775.782248:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828775.782252:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3325 00000100:00080000:3.0:1787828775.782255:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828775.782277:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828775.782288:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828775.782302:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280803968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828775.782313:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280803840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828775.782317:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828775.782323:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828775.782325:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828775.782330:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3325 00000100:00080000:3.0:1787828775.782335:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828775.782338:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828775.782342:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828775.782357:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280804096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828775.782366:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828775.782369:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828775.782371:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828775.782374:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3325 00000100:00080000:3.0:1787828775.782378:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828775.782381:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828775.782387:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828775.782409:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828775.782433:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280804224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828775.782442:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828775.782445:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828775.782447:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828775.782451:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3325 00000100:00080000:3.0:1787828775.782455:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828775.782460:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828775/real 1787828775] req@ffff8ab3d0a412c0 x1874671280803840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828775.782461:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828775.782468:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828775.782487:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280804352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828775.782494:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280803840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828775.782531:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828775.782538:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828775.782544:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280803840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828775.782576:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280803968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828775.782599:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828775.782612:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828775/real 1787828775] req@ffff8ab3d0a421c0 x1874671280803968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828775.782622:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280803968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828775.782629:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828775.782632:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828775.782635:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280803968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828775.782640:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280804096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828775.782653:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828775.782660:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828775/real 1787828775] req@ffff8ab3d0a40000 x1874671280804096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828775.782669:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280804096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828775.782675:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828775.782677:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828775.782680:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280804096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828775.782684:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280804224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828775.782700:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828775.782709:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828775/real 1787828775] req@ffff8ab3d0a43840 x1874671280804224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828775.782721:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280804224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828775.782728:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828775.782731:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828775.782733:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280804224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828775.782738:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280804352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828775.782749:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828775.782756:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828775/real 1787828775] req@ffff8ab3d0a43480 x1874671280804352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828775.782764:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280804352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828775.782774:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828775.782777:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828775.782779:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280804352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787828776.806055:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828777.830073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828778.854066:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828779.878071:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828780.902017:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828780.902027:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828780.902032:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828780.902042:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3330 00010000:00010000:1.0:1787828780.902044:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828780.902052:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828780.902077:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828780.902122:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280804480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828780.902162:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828780.902166:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828780.902168:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828780.902174:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3330 00000100:00080000:3.0:1787828780.902179:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828780.902184:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828780.902193:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280804480:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787828780.902194:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828780.902213:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280804608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828780.902225:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828780.902228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828780.902231:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828780.902236:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3330 00000100:00080000:3.0:1787828780.902251:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828780.902257:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828780.902262:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787828780.902266:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828780.902283:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280804736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828780.902294:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828780.902297:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828780.902299:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828780.902304:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3330 00000100:00080000:3.0:1787828780.902308:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828780.902312:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828780/real 1787828780] req@ffff8ab3d0a41e00 x1874671280804480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828780.902315:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828780.902326:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828780.902333:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280804480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828780.902343:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828780.902343:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280804864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828780.902351:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828780.902358:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280804480:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828780.902365:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828780.902370:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828780.902372:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828780.902377:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3330 00000100:00080000:3.0:1787828780.902381:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828780.902385:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828780.902390:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828780.902395:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280804608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787828780.902405:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280804992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828780.902426:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828780.902439:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828780/real 1787828780] req@ffff8ab3d0a430c0 x1874671280804608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828780.902449:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280804608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828780.902460:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828780.902464:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828780.902468:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280804608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828780.902474:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280804736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828780.902495:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828780.902507:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828780/real 1787828780] req@ffff8ab3d0a43c00 x1874671280804736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828780.902520:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280804736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828780.902536:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828780.902539:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828780.902542:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280804736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828780.902546:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280804864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828780.902562:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828780.902585:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828780/real 1787828780] req@ffff8ab3d0a43480 x1874671280804864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828780.902593:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280804864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828780.902599:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828780.902602:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828780.902604:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280804864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828780.902615:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280804992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828780.902626:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828780.902632:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828780/real 1787828780] req@ffff8ab3d0a43840 x1874671280804992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828780.902640:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280804992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828780.902646:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828780.902648:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828780.902650:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280804992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828781.926155:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828782.950021:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828783.974054:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828784.998024:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828786.022012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828786.022024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828786.022029:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828786.022042:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3335 00000100:00080000:1.0:1787828786.022055:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828786.022064:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828786.022132:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280805120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828786.022192:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828786.022196:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828786.022198:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828786.022201:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3335 00000100:00080000:1.0:1787828786.022205:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828786.022209:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828786.022222:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828786.022236:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280805248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828786.022247:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828786.022251:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280805120:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787828786.022252:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828786.022254:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828786.022274:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3335 00000100:00080000:1.0:1787828786.022278:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828786.022281:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828786.022286:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828786.022296:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280805376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828786.022306:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828786.022310:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828786.022312:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828786.022318:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3335 00000100:00080000:1.0:1787828786.022322:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828786.022340:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828786.022347:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828786.022362:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280805504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828786.022375:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828786.022380:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828786.022382:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828786.022388:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3335 00000100:00080000:1.0:1787828786.022391:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828786.022395:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828786.022400:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828786.022410:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280805632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828786.022526:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00010000:00010000:2.0:1787828786.022547:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828786.022571:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828786/real 1787828786] req@ffff8ab3d22ff480 x1874671280805120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828786.022596:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280805120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828786.022605:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828786.022608:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828786.022611:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280805120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828786.022634:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280805248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828786.022651:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828786.022659:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828786/real 1787828786] req@ffff8ab3d22fc000 x1874671280805248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828786.022667:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280805248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828786.022672:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828786.022675:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828786.022677:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280805248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828786.022682:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280805376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828786.022693:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828786.022699:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828786/real 1787828786] req@ffff8ab3d22fda40 x1874671280805376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828786.022706:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280805376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828786.022711:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828786.022713:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828786.022716:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280805376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828786.022720:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280805504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828786.022730:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828786.022736:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828786/real 1787828786] req@ffff8ab3d22fcb40 x1874671280805504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828786.022743:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280805504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828786.022750:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828786.022752:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828786.022754:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280805504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828786.022757:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280805632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828786.022767:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828786.022773:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828786/real 1787828786] req@ffff8ab3d22fd680 x1874671280805632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828786.022780:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280805632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828786.022786:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828786.022787:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828786.022789:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280805632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787828787.046035:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828788.070184:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828789.094054:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828790.118120:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828791.141984:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828791.141993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828791.141996:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828791.142004:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3340 00000100:00080000:1.0:1787828791.142054:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828791.142063:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828791.142122:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280805760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828791.142147:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828791.142150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828791.142151:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828791.142154:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3340 00000100:00080000:1.0:1787828791.142157:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828791.142160:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828791.142173:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828791.142183:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280805888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828791.142191:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828791.142193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828791.142195:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828791.142198:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3340 00000100:00080000:1.0:1787828791.142201:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828791.142203:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828791.142207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828791.142211:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280805760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828791.142217:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280806016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828791.142228:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828791.142231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828791.142234:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828791.142238:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3340 00000100:00080000:1.0:1787828791.142242:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828791.142247:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828791.142252:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828791.142265:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280806144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828791.142275:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828791.142278:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828791.142280:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828791.142284:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3340 00000100:00080000:1.0:1787828791.142291:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828791.142297:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828791.142302:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:0.0:1787828791.142303:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828791.142319:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280806272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828791.142372:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828791/real 1787828791] req@ffff8ab3d22fc780 x1874671280805760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1787828791.142384:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828791.142394:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280805760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828791.142407:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828791.142412:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828791.142424:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280805760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828791.142432:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280805888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828791.142455:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828791.142468:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828791/real 1787828791] req@ffff8ab3d22fe1c0 x1874671280805888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828791.142482:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280805888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828791.142497:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828791.142501:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828791.142504:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280805888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828791.142526:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280806016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828791.142542:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828791.142552:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828791/real 1787828791] req@ffff8ab3d22fde00 x1874671280806016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828791.142564:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280806016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828791.142581:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828791.142584:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828791.142588:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280806016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828791.142594:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280806144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828791.142611:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828791.142620:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828791/real 1787828791] req@ffff8ab3d22fd2c0 x1874671280806144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828791.142628:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280806144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828791.142634:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828791.142637:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828791.142639:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280806144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828791.142643:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280806272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828791.142654:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828791.142659:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828791/real 1787828791] req@ffff8ab3d22fed00 x1874671280806272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828791.142666:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280806272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828791.142672:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828791.142674:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828791.142676:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280806272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828792.166080:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828793.190001:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828794.214082:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828795.238029:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828796.261997:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828796.262005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828796.262009:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1787828796.262016:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828796.262019:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3345 00000100:00080000:3.0:1787828796.262028:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828796.262051:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828796.262086:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280806400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828796.262115:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828796.262119:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828796.262121:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828796.262125:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3345 00000100:00080000:3.0:1787828796.262129:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828796.262134:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828796.262144:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828796.262158:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280806528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828796.262165:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280806400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828796.262177:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828796.262182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828796.262184:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828796.262189:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3345 00000100:00080000:3.0:1787828796.262193:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828796.262198:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828796.262204:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828796.262218:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280806656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828796.262239:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828796.262242:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828796.262245:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828796.262249:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3345 00000100:00080000:3.0:1787828796.262254:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828796.262258:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828796.262263:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787828796.262269:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828796.262278:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280806784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828796.262289:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828796.262293:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828796.262295:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828796.262300:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3345 00000100:00080000:3.0:1787828796.262305:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828796.262310:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828796.262314:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828796/real 1787828796] req@ffff8ab3d0a40000 x1874671280806400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828796.262318:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828796.262337:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280806912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828796.262338:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280806400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828796.262370:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828796.262375:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828796.262379:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280806400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828796.262399:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280806528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828796.262427:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828796.262436:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828796/real 1787828796] req@ffff8ab3d0a421c0 x1874671280806528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828796.262444:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280806528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828796.262450:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828796.262453:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828796.262455:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280806528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828796.262459:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280806656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828796.262470:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828796.262475:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828796/real 1787828796] req@ffff8ab3d0a412c0 x1874671280806656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828796.262482:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280806656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828796.262488:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828796.262489:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828796.262491:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280806656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828796.262495:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280806784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828796.262505:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828796.262510:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828796/real 1787828796] req@ffff8ab3d0a43840 x1874671280806784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828796.262517:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280806784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828796.262524:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828796.262526:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828796.262528:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280806784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828796.262531:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280806912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828796.262540:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828796.262545:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828796/real 1787828796] req@ffff8ab3d0a43480 x1874671280806912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828796.262551:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280806912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828796.262557:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828796.262559:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828796.262561:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280806912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787828797.286020:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828798.310114:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828799.333987:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828800.358117:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828801.381992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828801.382014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828801.382017:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1787828801.382028:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828801.382028:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3351 00000100:00080000:3.0:1787828801.382061:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828801.382068:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828801.382102:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280807040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828801.382140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828801.382143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828801.382145:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828801.382148:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3351 00000100:00080000:3.0:1787828801.382151:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828801.382154:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828801.382162:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828801.382170:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280807040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828801.382174:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280807168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828801.382184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828801.382186:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828801.382188:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828801.382213:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3351 00000100:00080000:3.0:1787828801.382216:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828801.382219:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828801.382224:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828801.382235:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280807296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828801.382244:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828801.382247:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828801.382248:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828801.382252:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3351 00000800:00000100:0.0:1787828801.382258:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828801.382258:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828801.382263:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828801.382267:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828801.382279:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280807424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828801.382287:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828801.382308:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828801.382310:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828801.382315:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3351 00000100:00080000:3.0:1787828801.382320:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828801.382325:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828801.382330:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828801.382331:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828801/real 1787828801] req@ffff8ab3d0a43c00 x1874671280807040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828801.382347:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280807552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828801.382357:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280807040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828801.382371:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828801.382375:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828801.382378:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280807040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828801.382398:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280807168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828801.382417:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828801.382428:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828801/real 1787828801] req@ffff8ab3d0a430c0 x1874671280807168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828801.382436:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280807168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828801.382442:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828801.382445:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828801.382447:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280807168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828801.382468:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280807296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828801.382482:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828801.382488:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828801/real 1787828801] req@ffff8ab3d0a41e00 x1874671280807296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828801.382497:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280807296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828801.382503:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828801.382514:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828801.382534:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280807296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828801.382539:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280807424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828801.382566:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828801.382576:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828801/real 1787828801] req@ffff8ab3d0a43480 x1874671280807424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828801.382585:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280807424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828801.382593:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828801.382595:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828801.382598:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280807424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828801.382602:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280807552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828801.382622:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828801.382630:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828801/real 1787828801] req@ffff8ab3d0a43840 x1874671280807552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828801.382639:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280807552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828801.382646:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828801.382665:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828801.382667:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280807552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828802.406073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828803.430034:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828804.454086:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828805.478100:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828806.502045:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828806.502057:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828806.502061:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828806.502072:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3356 00000100:00080000:2.0:1787828806.502083:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828806.502092:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828806.502095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828806.502135:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280807680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828806.502164:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828806.502168:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828806.502170:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828806.502174:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3356 00000100:00080000:2.0:1787828806.502178:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828806.502181:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828806.502191:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828806.502214:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280807808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828806.502226:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828806.502230:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828806.502233:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828806.502239:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280807680:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828806.502240:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3356 00000100:00080000:2.0:1787828806.502247:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828806.502257:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828806.502275:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828806.502295:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280807936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828806.502312:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828806.502318:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828806.502321:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828806.502328:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3356 00000100:00080000:2.0:1787828806.502335:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828806.502343:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828806.502345:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828806.502355:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828806.502372:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280808064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828806.502382:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828806.502386:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828806.502389:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828806.502394:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3356 00000100:00080000:2.0:1787828806.502400:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828806.502407:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828806/real 1787828806] req@ffff8ab3f0439a40 x1874671280807680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828806.502408:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828806.502418:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828806.502437:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280808192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828806.502443:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280807680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828806.502464:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828806.502469:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828806.502476:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280807680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828806.502501:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280807808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828806.502535:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828806.502547:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828806/real 1787828806] req@ffff8ab3f043a940 x1874671280807808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828806.502559:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280807808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828806.502568:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828806.502571:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828806.502574:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280807808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828806.502580:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280807936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787828806.502593:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828806.502601:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828806/real 1787828806] req@ffff8ab3f043a1c0 x1874671280807936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828806.502612:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280807936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828806.502620:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828806.502622:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828806.502626:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280807936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828806.502631:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280808064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828806.502646:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828806.502653:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828806/real 1787828806] req@ffff8ab3f0439e00 x1874671280808064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828806.502682:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280808064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828806.502691:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828806.502694:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828806.502697:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280808064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787828806.502705:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280808192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787828806.502732:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828806.502749:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828806/real 1787828806] req@ffff8ab3f043b480 x1874671280808192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828806.502787:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280808192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828806.502806:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828806.502809:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828806.502813:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280808192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787828807.526075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828808.550063:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828809.574078:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828810.598141:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828811.622042:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828811.622054:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828811.622060:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828811.622084:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3361 00000100:00080000:0.0:1787828811.622096:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1787828811.622100:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828811.622105:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828811.622157:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280808320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828811.622186:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828811.622189:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828811.622191:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828811.622194:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3361 00000100:00080000:0.0:1787828811.622197:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828811.622200:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828811.622223:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828811.622231:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280808320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828811.622237:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280808448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828811.622250:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828811.622253:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828811.622254:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828811.622257:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3361 00000100:00080000:0.0:1787828811.622259:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828811.622262:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828811.622265:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828811.622274:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280808576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828811.622286:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828811.622290:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828811.622291:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828811.622294:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3361 00000100:00080000:0.0:1787828811.622308:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828811.622311:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828811.622315:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828811.622321:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828811.622325:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280808704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828811.622333:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828811.622336:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828811.622338:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828811.622341:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3361 00000100:00080000:0.0:1787828811.622344:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828811.622347:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828811.622350:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828811.622358:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280808832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828811.622400:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828811/real 1787828811] req@ffff8ab3f075e580 x1874671280808320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828811.622420:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280808320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828811.622430:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828811.622440:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828811.622444:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280808320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828811.622470:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280808448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828811.622500:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828811.622511:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828811/real 1787828811] req@ffff8ab3f075f480 x1874671280808448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828811.622526:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280808448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828811.622533:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828811.622535:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828811.622538:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280808448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828811.622542:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280808576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828811.622573:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828811.622586:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828811/real 1787828811] req@ffff8ab3f075da40 x1874671280808576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828811.622600:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280808576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828811.622611:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828811.622615:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828811.622619:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280808576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828811.622625:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280808704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828811.622644:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828811.622652:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828811/real 1787828811] req@ffff8ab3f075ed00 x1874671280808704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828811.622660:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280808704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828811.622674:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828811.622676:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828811.622678:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280808704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828811.622682:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280808832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828811.622704:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828811.622710:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828811/real 1787828811] req@ffff8ab3f075e940 x1874671280808832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828811.622717:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280808832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828811.622725:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828811.622727:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828811.622729:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280808832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828812.646170:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828813.670008:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828814.694145:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828815.718094:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828816.742036:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828816.742063:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828816.742068:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828816.742079:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3366 00000100:00080000:0.0:1787828816.742091:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828816.742099:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828816.742119:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828816.742151:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280808960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828816.742197:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828816.742202:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828816.742205:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828816.742212:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3366 00000100:00080000:0.0:1787828816.742219:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828816.742243:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828816.742256:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828816.742257:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280808960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828816.742274:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280809088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828816.742287:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828816.742291:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828816.742293:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828816.742297:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3366 00000100:00080000:0.0:1787828816.742344:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828816.742348:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828816.742354:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828816.742366:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280809216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828816.742375:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828816.742379:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828816.742381:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828816.742386:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3366 00000100:00080000:0.0:1787828816.742389:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828816.742394:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828816.742408:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828816.742420:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280809344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828816.742426:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280809088:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787828816.742459:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828816.742463:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828816.742466:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828816.742472:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3366 00000100:00080000:0.0:1787828816.742478:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828816.742482:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828816.742488:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828816.742489:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280809216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787828816.742510:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280809472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828816.742544:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280809344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828816.742569:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280809472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787828817.766026:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828818.790081:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828819.814114:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828819.814331:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787828819.814413:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828819.814435:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828819.814513:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828819.814527:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828819.814547:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828819.814558:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:2.0:1787828819.814633:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828816/real 1787828819] req@ffff8ab3f075f840 x1874671280809344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828819.814660:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280809344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828819.814677:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828819.814682:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828819.814687:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280809344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828819.814696:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828816/real 1787828819] req@ffff8ab3f075cb40 x1874671280809472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828819.814714:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280809472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828819.814726:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828819.814730:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828819.814733:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280809472:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828819.814740:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828816/real 1787828819] req@ffff8ab3f075f0c0 x1874671280809088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828819.814754:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280809088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828819.814770:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828819.814774:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828819.814778:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280809088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828819.814785:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828816/real 1787828819] req@ffff8ab3f075de00 x1874671280809216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828819.814799:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280809216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828819.814808:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828819.814812:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828819.814816:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280809216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828819.814823:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828816/real 1787828819] req@ffff8ab3f075cf00 x1874671280808960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828819.814837:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280808960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828819.814846:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828819.814849:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828819.814852:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280808960:192.168.203.124@tcp:250:kworker.0 00010000:00010000:3.0:1787828820.838078:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828821.862022:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828821.862067:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828821.862071:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828821.862082:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3371 00000100:00080000:0.0:1787828821.862091:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828821.862098:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828821.862117:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828821.862140:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280809600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828821.862162:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828821.862170:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828821.862171:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828821.862175:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3371 00000100:00080000:0.0:1787828821.862178:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828821.862182:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828821.862191:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828821.862198:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280809600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828821.862204:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280809728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828821.862214:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828821.862217:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828821.862219:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828821.862222:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3371 00000100:00080000:0.0:1787828821.862225:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828821.862227:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828821.862231:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828821.862259:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280809856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828821.862267:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828821.862269:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828821.862270:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828821.862273:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3371 00000100:00080000:0.0:1787828821.862275:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828821.862278:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828821.862281:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828821.862292:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280809984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828821.862302:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828821.862306:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828821.862308:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828821.862315:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828821.862316:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3371 00000100:00080000:0.0:1787828821.862324:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828821.862330:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828821.862336:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828821.862352:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280810112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828821.862363:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828821/real 1787828821] req@ffff8ab3f075c000 x1874671280809600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828821.862385:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280809600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828821.862399:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828821.862404:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828821.862409:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280809600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828821.862444:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280809728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828821.862472:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828821.862489:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828821/real 1787828821] req@ffff8ab3f075cf00 x1874671280809728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828821.862506:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280809728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828821.862515:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828821.862518:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828821.862522:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280809728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828821.862529:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280809856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828821.862552:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828821.862566:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828821/real 1787828821] req@ffff8ab3f075de00 x1874671280809856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828821.862580:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280809856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828821.862596:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828821.862600:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828821.862603:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280809856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828821.862609:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280809984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828821.862630:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828821.862644:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828821/real 1787828821] req@ffff8ab3f075f0c0 x1874671280809984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828821.862657:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280809984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828821.862667:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828821.862671:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828821.862675:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280809984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828821.862682:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280810112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828821.862703:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828821.862715:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828821/real 1787828821] req@ffff8ab3f075cb40 x1874671280810112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828821.862728:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280810112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828821.862740:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828821.862744:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828821.862749:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280810112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828822.886197:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828823.910075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828824.934131:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828825.958021:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828826.982077:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828826.982091:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828826.982097:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828826.982112:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3376 00000100:00080000:3.0:1787828826.982127:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828826.982139:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828826.982173:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828826.982194:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280810240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828826.982235:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828826.982240:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828826.982243:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828826.982248:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3376 00000100:00080000:3.0:1787828826.982254:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828826.982261:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828826.982273:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828826.982280:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280810240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828826.982295:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280810368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828826.982312:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828826.982317:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828826.982320:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828826.982325:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3376 00000100:00080000:3.0:1787828826.982348:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828826.982353:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828826.982360:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828826.982361:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828826.982383:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280810496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828826.982396:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828826.982400:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828826.982403:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828826.982408:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3376 00000100:00080000:3.0:1787828826.982411:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828826.982415:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828826.982422:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828826.982440:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280810624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828826.982447:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828826/real 1787828826] req@ffff8ab3d0a412c0 x1874671280810240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828826.982473:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828826.982498:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828826.982500:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828826.982504:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3376 00000100:00080000:3.0:1787828826.982507:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828826.982510:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280810240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828826.982513:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828826.982520:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828826.982532:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280810752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828826.982533:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828826.982546:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828826.982553:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280810240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828826.982576:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280810368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828826.982594:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828826.982605:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828826/real 1787828826] req@ffff8ab3d0a421c0 x1874671280810368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828826.982614:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280810368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828826.982637:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828826.982641:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828826.982646:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280810368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828826.982653:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280810496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828826.982677:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828826.982688:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828826/real 1787828826] req@ffff8ab3d0a40000 x1874671280810496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828826.982702:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280810496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828826.982711:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828826.982714:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828826.982717:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280810496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828826.982723:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280810624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828826.982741:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828826.982751:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828826/real 1787828826] req@ffff8ab3d0a43840 x1874671280810624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828826.982764:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280810624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828826.982774:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828826.982777:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828826.982780:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280810624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828826.982786:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280810752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828826.982802:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828826.982850:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828826/real 1787828826] req@ffff8ab3d0a43480 x1874671280810752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828826.982870:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280810752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828826.982880:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828826.982883:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828826.982936:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280810752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828828.006003:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828829.030053:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828830.054035:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828831.078145:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828832.102094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828832.102116:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828832.102120:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828832.102134:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3381 00000100:00080000:0.0:1787828832.102147:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828832.102158:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828832.102175:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828832.102208:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280810880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828832.102259:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828832.102265:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828832.102269:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828832.102275:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3381 00000100:00080000:0.0:1787828832.102281:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828832.102288:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828832.102303:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828832.102336:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280811008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828832.102350:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280810880:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828832.102353:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828832.102359:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828832.102362:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828832.102368:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3381 00000100:00080000:0.0:1787828832.102373:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828832.102378:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828832.102386:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828832.102401:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280811136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828832.102425:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828832.102429:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828832.102430:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828832.102434:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828832.102439:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3381 00000100:00080000:0.0:1787828832.102444:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828832.102451:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828832.102458:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828832.102474:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280811264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828832.102488:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828832.102494:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828832.102496:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828832.102502:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3381 00000100:00080000:0.0:1787828832.102507:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828832.102513:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828832.102517:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828832/real 1787828832] req@ffff8ab3f075f840 x1874671280810880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828832.102521:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828832.102535:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280811392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828832.102546:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280810880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828832.102574:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828832.102582:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828832.102589:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280810880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828832.102624:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280811008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828832.102668:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828832.102685:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828832/real 1787828832] req@ffff8ab3f075e940 x1874671280811008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828832.102699:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280811008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828832.102709:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828832.102713:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828832.102717:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280811008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828832.102723:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280811136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828832.102739:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828832.102749:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828832/real 1787828832] req@ffff8ab3f075ed00 x1874671280811136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828832.102761:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280811136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828832.102769:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828832.102773:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828832.102776:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280811136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828832.102781:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280811264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828832.102840:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828832.102879:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828832/real 1787828832] req@ffff8ab3f075da40 x1874671280811264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828832.102894:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280811264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828832.102951:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828832.102955:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828832.102972:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280811264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828832.102980:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280811392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828832.103006:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828832.103021:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828832/real 1787828832] req@ffff8ab3f075f480 x1874671280811392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828832.103036:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280811392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828832.103047:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828832.103051:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828832.103055:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280811392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828833.126093:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828834.150095:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828835.174129:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828836.198071:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828837.221994:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828837.222004:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828837.222009:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828837.222019:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3386 00000100:00080000:3.0:1787828837.222031:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828837.222039:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828837.222040:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828837.222080:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280811520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828837.222110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828837.222114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828837.222116:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828837.222121:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3386 00000100:00080000:3.0:1787828837.222125:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828837.222130:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828837.222139:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280811520:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787828837.222140:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828837.222160:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280811648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828837.222172:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828837.222176:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828837.222178:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828837.222182:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3386 00000100:00080000:3.0:1787828837.222187:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828837.222191:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787828837.222194:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787828837.222199:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828837.222215:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280811776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828837.222226:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828837.222230:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828837.222232:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828837.222236:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828837/real 1787828837] req@ffff8ab3d0a41e00 x1874671280811520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828837.222241:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3386 00000100:00080000:3.0:1787828837.222249:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828837.222255:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828837.222259:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280811520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828837.222262:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:2.0:1787828837.222274:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828837.222277:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828837.222283:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280811520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828837.222284:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280811904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828837.222300:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828837.222304:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828837.222306:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828837.222310:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280811648:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787828837.222313:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3386 00000100:00080000:3.0:1787828837.222321:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828837.222327:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787828837.222332:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787828837.222337:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828837.222344:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828837/real 1787828837] req@ffff8ab3d0a430c0 x1874671280811648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828837.222355:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280812032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828837.222356:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280811648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828837.222387:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828837.222391:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828837.222394:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280811648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828837.222399:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280811776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828837.222412:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828837.222419:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828837/real 1787828837] req@ffff8ab3d0a43c00 x1874671280811776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828837.222427:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280811776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828837.222432:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828837.222434:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828837.222436:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280811776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828837.222440:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280811904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828837.222450:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828837.222456:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828837/real 1787828837] req@ffff8ab3d0a43480 x1874671280811904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828837.222464:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280811904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828837.222474:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828837.222477:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828837.222481:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280811904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828837.222495:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280812032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828837.222516:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828837.222528:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828837/real 1787828837] req@ffff8ab3d0a43840 x1874671280812032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828837.222538:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280812032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828837.222545:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828837.222547:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828837.222549:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280812032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828838.246055:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828839.270109:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828840.294111:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828841.318031:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828842.342013:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828842.342021:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828842.342025:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1787828842.342034:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828842.342035:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3391 00000100:00080000:0.0:1787828842.342047:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828842.342053:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828842.342086:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280812160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828842.342113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828842.342116:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828842.342117:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828842.342120:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3391 00000100:00080000:0.0:1787828842.342123:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828842.342126:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828842.342134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828842.342145:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280812288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828842.342154:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828842.342160:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828842.342163:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828842.342163:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280812160:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828842.342178:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3391 00000100:00080000:0.0:1787828842.342181:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828842.342184:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828842.342187:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828842.342198:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280812416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828842.342204:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828842.342207:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828842.342209:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828842.342212:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3391 00000100:00080000:0.0:1787828842.342214:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828842.342217:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828842.342229:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828842.342240:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280812544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828842.342247:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828842.342251:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787828842.342254:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828842.342256:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828842.342260:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3391 00000100:00080000:0.0:1787828842.342265:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828842.342269:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828842.342275:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828842.342285:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280812672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828842.342323:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828842/real 1787828842] req@ffff8ab3f075e580 x1874671280812160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828842.342345:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280812160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828842.342354:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828842.342357:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828842.342361:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280812160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828842.342384:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280812288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828842.342401:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828842.342410:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828842/real 1787828842] req@ffff8ab3f075f480 x1874671280812288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828842.342418:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280812288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828842.342423:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828842.342425:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828842.342428:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280812288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828842.342432:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280812416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828842.342442:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828842.342447:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828842/real 1787828842] req@ffff8ab3f075da40 x1874671280812416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828842.342455:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280812416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828842.342464:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828842.342466:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828842.342468:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280812416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828842.342472:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280812544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828842.342483:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828842.342489:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828842/real 1787828842] req@ffff8ab3f075ed00 x1874671280812544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828842.342496:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280812544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828842.342503:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828842.342504:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828842.342506:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280812544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828842.342510:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280812672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828842.342520:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828842.342525:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828842/real 1787828842] req@ffff8ab3f075e940 x1874671280812672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828842.342532:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280812672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828842.342538:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828842.342540:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828842.342542:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280812672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828843.366140:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828844.390042:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828845.414037:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828846.438191:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828847.462006:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828847.462022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828847.462028:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828847.462043:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3397 00000100:00080000:0.0:1787828847.462055:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828847.462064:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828847.462113:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828847.462117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280812800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828847.462139:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828847.462142:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828847.462144:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828847.462147:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3397 00000100:00080000:0.0:1787828847.462150:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828847.462153:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828847.462165:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828847.462165:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280812800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828847.462180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280812928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828847.462189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828847.462192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828847.462194:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828847.462198:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3397 00000100:00080000:0.0:1787828847.462200:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828847.462203:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828847.462207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828847.462216:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280813056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828847.462223:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828847.462225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828847.462227:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828847.462229:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3397 00000100:00080000:0.0:1787828847.462233:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828847.462236:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828847.462240:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787828847.462246:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828847.462250:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280813184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828847.462258:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828847.462261:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828847.462263:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828847.462266:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3397 00000100:00080000:0.0:1787828847.462268:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828847.462271:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828847.462274:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828847.462281:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280813312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828847.462311:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828847/real 1787828847] req@ffff8ab3f075f840 x1874671280812800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828847.462337:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280812800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828847.462352:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828847.462357:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828847.462364:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280812800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828847.462394:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280812928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828847.462419:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828847.462435:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828847/real 1787828847] req@ffff8ab3f075cb40 x1874671280812928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828847.462448:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280812928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828847.462457:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828847.462461:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828847.462464:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280812928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828847.462474:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280813056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828847.462490:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828847.462499:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828847/real 1787828847] req@ffff8ab3f075f0c0 x1874671280813056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828847.462506:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280813056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828847.462512:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828847.462514:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828847.462516:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280813056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828847.462520:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280813184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828847.462530:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828847.462536:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828847/real 1787828847] req@ffff8ab3f075de00 x1874671280813184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828847.462543:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280813184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828847.462549:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828847.462551:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828847.462553:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280813184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828847.462556:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280813312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828847.462566:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828847.462571:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828847/real 1787828847] req@ffff8ab3f075cf00 x1874671280813312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828847.462579:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280813312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828847.462586:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828847.462587:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828847.462589:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280813312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828848.486046:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828849.510028:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828850.534075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828851.558077:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828852.582034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828852.582046:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828852.582052:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828852.582064:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3402 00000100:00080000:0.0:1787828852.582076:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828852.582086:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828852.582134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280813440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828852.582163:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828852.582170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828852.582180:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828852.582184:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828852.582190:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3402 00000100:00080000:0.0:1787828852.582195:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828852.582201:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828852.582212:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828852.582230:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280813568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828852.582249:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280813440:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828852.582255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828852.582284:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828852.582288:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828852.582298:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3402 00000100:00080000:0.0:1787828852.582307:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828852.582316:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828852.582328:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828852.582356:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280813696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828852.582358:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828852.582379:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828852.582383:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828852.582386:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828852.582395:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3402 00000100:00080000:0.0:1787828852.582401:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828852.582408:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828852.582417:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828852.582441:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828852/real 1787828852] req@ffff8ab3f075c000 x1874671280813440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828852.582442:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280813824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828852.582513:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828852.582519:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828852.582522:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828852.582533:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3402 00000100:00100000:2.0:1787828852.582535:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280813440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828852.582541:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828852.582558:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:2.0:1787828852.582569:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828852.582571:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:2.0:1787828852.582582:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828852.582591:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280813440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828852.582599:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280813952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828852.582630:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280813568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828852.582668:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828852.582680:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828852/real 1787828852] req@ffff8ab3f075cf00 x1874671280813568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828852.582688:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280813568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828852.582695:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828852.582698:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828852.582700:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280813568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828852.582705:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280813696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828852.582717:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828852.582723:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828852/real 1787828852] req@ffff8ab3f075de00 x1874671280813696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828852.582730:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280813696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828852.582736:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828852.582738:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828852.582740:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280813696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828852.582744:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280813824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828852.582755:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828852.582761:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828852/real 1787828852] req@ffff8ab3f075f0c0 x1874671280813824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828852.582768:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280813824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828852.582776:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828852.582778:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828852.582780:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280813824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828852.582787:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280813952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828852.582806:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828852.582819:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828852/real 1787828852] req@ffff8ab3f075cb40 x1874671280813952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828852.582832:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280813952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828852.582842:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828852.582845:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828852.582847:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280813952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828853.606127:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828854.630052:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828855.654163:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828856.678121:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828857.702005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828857.702017:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828857.702023:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828857.702051:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3407 00000100:00080000:0.0:1787828857.702064:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828857.702071:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828857.702090:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828857.702117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280814080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828857.702141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828857.702144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828857.702145:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828857.702149:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3407 00000100:00080000:0.0:1787828857.702152:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828857.702156:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828857.702167:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828857.702179:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280814080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828857.702180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280814208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828857.702195:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828857.702197:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828857.702199:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828857.702202:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3407 00000100:00080000:0.0:1787828857.702204:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828857.702207:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828857.702210:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828857.702220:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280814336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828857.702227:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828857.702230:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828857.702232:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828857.702236:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3407 00000100:00080000:0.0:1787828857.702257:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828857.702263:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787828857.702270:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787828857.702271:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828857.702288:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280814464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828857.702303:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828857.702306:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828857.702308:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828857.702311:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3407 00000100:00080000:0.0:1787828857.702330:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828857.702335:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828857.702341:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828857.702355:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280814592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828857.702363:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828857/real 1787828857] req@ffff8ab3f075f840 x1874671280814080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828857.702396:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280814080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828857.702413:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828857.702421:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828857.702431:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280814080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828857.702474:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280814208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828857.702501:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828857.702514:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828857/real 1787828857] req@ffff8ab3f075e940 x1874671280814208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828857.702525:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280814208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828857.702532:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828857.702535:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828857.702538:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280814208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828857.702543:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280814336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828857.702556:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828857.702564:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828857/real 1787828857] req@ffff8ab3f075ed00 x1874671280814336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828857.702573:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280814336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828857.702579:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828857.702582:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828857.702584:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280814336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828857.702588:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280814464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828857.702602:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828857.702631:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828857/real 1787828857] req@ffff8ab3f075da40 x1874671280814464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828857.702642:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280814464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828857.702651:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828857.702654:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828857.702657:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280814464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828857.702662:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280814592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828857.702678:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828857.702686:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828857/real 1787828857] req@ffff8ab3f075f480 x1874671280814592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828857.702696:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280814592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828857.702704:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828857.702706:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828857.702708:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280814592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828858.726065:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828859.750094:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828860.774195:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828861.798065:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828862.821994:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828862.822002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828862.822006:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828862.822015:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3412 00000100:00080000:0.0:1787828862.822024:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828862.822031:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828862.822065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280814720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787828862.822067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828862.822101:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828862.822106:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828862.822108:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828862.822113:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3412 00000100:00080000:0.0:1787828862.822118:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828862.822124:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828862.822134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828862.822147:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280814848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828862.822157:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828862.822161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828862.822163:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828862.822171:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3412 00000100:00100000:2.0:1787828862.822173:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280814720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828862.822175:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828862.822187:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828862.822194:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828862.822203:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280814976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828862.822209:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828862.822213:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828862.822214:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828862.822217:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3412 00000100:00080000:0.0:1787828862.822220:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828862.822222:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828862.822226:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828862.822233:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280815104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828862.822240:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828862.822243:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828862.822245:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828862.822249:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3412 00000100:00080000:0.0:1787828862.822263:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828862.822269:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828862.822275:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:2.0:1787828862.822279:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787828862.822295:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280815232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828862.822350:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828862/real 1787828862] req@ffff8ab3f075e580 x1874671280814720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828862.822395:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280814720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828862.822410:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828862.822415:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828862.822420:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280814720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828862.822485:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280814848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828862.822511:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828862.822525:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828862/real 1787828862] req@ffff8ab3f075f480 x1874671280814848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828862.822538:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280814848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828862.822547:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828862.822551:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828862.822555:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280814848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828862.822562:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280814976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828862.822599:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828862.822613:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828862/real 1787828862] req@ffff8ab3f075da40 x1874671280814976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828862.822636:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280814976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828862.822653:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828862.822657:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828862.822661:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280814976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828862.822668:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280815104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828862.822687:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828862.822699:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828862/real 1787828862] req@ffff8ab3f075ed00 x1874671280815104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828862.822712:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280815104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828862.822740:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828862.822743:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828862.822747:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280815104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828862.822754:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280815232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828862.822773:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828862.822783:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828862/real 1787828862] req@ffff8ab3f075e940 x1874671280815232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828862.822797:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280815232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828862.822808:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828862.822812:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828862.822816:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280815232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828863.846112:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828864.870045:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828865.894102:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828866.918137:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828867.942002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828867.942012:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828867.942017:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828867.942028:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3417 00010000:00010000:2.0:1787828867.942029:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828867.942038:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828867.942045:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828867.942085:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280815360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828867.942104:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828867.942107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828867.942109:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828867.942112:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3417 00000100:00080000:3.0:1787828867.942115:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828867.942119:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828867.942127:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828867.942132:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280815360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828867.942139:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280815488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828867.942148:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828867.942151:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828867.942152:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828867.942155:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3417 00000100:00080000:3.0:1787828867.942158:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828867.942160:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828867.942165:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828867.942177:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280815616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828867.942188:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828867.942193:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828867.942198:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828867.942201:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828867.942208:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3417 00000100:00080000:3.0:1787828867.942212:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828867.942217:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828867.942222:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828867.942236:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280815744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828867.942240:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828867/real 1787828867] req@ffff8ab3d0a40000 x1874671280815360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828867.942259:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828867.942264:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280815360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828867.942265:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828867.942269:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828867.942279:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3417 00000100:00080000:2.0:1787828867.942281:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828867.942284:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828867.942286:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828867.942296:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280815360:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828867.942296:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828867.942307:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828867.942322:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280815488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787828867.942326:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280815872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828867.942345:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828867.942358:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828867/real 1787828867] req@ffff8ab3d0a421c0 x1874671280815488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828867.942367:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280815488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828867.942372:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828867.942374:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828867.942377:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280815488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828867.942382:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280815616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828867.942393:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828867.942398:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828867/real 1787828867] req@ffff8ab3d0a412c0 x1874671280815616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828867.942406:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280815616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828867.942411:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828867.942413:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828867.942415:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280815616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828867.942418:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280815744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828867.942428:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828867.942434:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828867/real 1787828867] req@ffff8ab3d0a43840 x1874671280815744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828867.942441:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280815744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828867.942447:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828867.942448:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828867.942451:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280815744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828867.942465:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280815872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828867.942485:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828867.942497:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828867/real 1787828867] req@ffff8ab3d0a43480 x1874671280815872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828867.942507:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280815872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828867.942514:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828867.942516:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828867.942518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280815872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828868.966027:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828869.990031:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828871.014074:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828872.038073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828873.061976:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828873.061984:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828873.061987:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828873.061995:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3422 00000100:00080000:3.0:1787828873.062005:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828873.062012:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828873.062049:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828873.062051:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280816000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828873.062081:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828873.062085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828873.062087:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828873.062092:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3422 00000100:00080000:3.0:1787828873.062096:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828873.062100:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828873.062109:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280816000:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787828873.062110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828873.062122:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280816128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828873.062132:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828873.062135:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828873.062137:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828873.062140:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3422 00000100:00080000:3.0:1787828873.062156:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828873.062160:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828873.062164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828873.062165:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828873.062175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280816256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828873.062183:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828873.062187:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828873.062188:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828873.062192:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3422 00000100:00080000:3.0:1787828873.062195:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828873.062199:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828873.062204:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828873.062209:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828873/real 1787828873] req@ffff8ab3d0a43c00 x1874671280816000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828873.062214:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280816384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828873.062223:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828873.062226:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280816000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828873.062227:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828873.062229:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828873.062233:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3422 00000100:00080000:3.0:1787828873.062236:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828873.062237:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828873.062241:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828873.062245:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280816000:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828873.062248:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828873.062252:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828873.062262:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280816512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828873.062266:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280816128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828873.062289:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828873.062310:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828873/real 1787828873] req@ffff8ab3d0a430c0 x1874671280816128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828873.062327:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280816128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828873.062332:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828873.062335:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828873.062337:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280816128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828873.062341:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280816256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828873.062357:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828873.062364:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828873/real 1787828873] req@ffff8ab3d0a41e00 x1874671280816256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828873.062371:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280816256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828873.062376:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828873.062378:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828873.062380:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280816256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828873.062383:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280816384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828873.062394:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828873.062399:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828873/real 1787828873] req@ffff8ab3d0a43480 x1874671280816384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828873.062406:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280816384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828873.062417:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828873.062419:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828873.062421:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280816384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828873.062432:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280816512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828873.062441:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828873.062447:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828873/real 1787828873] req@ffff8ab3d0a43840 x1874671280816512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828873.062455:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280816512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828873.062461:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828873.062462:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828873.062464:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280816512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828874.086070:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828875.110106:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828876.134092:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828877.158059:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828878.182011:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828878.182023:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828878.182028:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828878.182039:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3427 00010000:00010000:2.0:1787828878.182040:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828878.182049:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828878.182056:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828878.182089:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280816640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828878.182109:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828878.182113:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828878.182116:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828878.182121:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3427 00000100:00080000:0.0:1787828878.182126:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828878.182131:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828878.182134:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280816640:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787828878.182142:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828878.182161:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280816768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828878.182177:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828878.182180:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828878.182182:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828878.182189:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3427 00000800:00000100:2.0:1787828878.182191:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828878.182198:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828878.182206:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828878.182212:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828878.182228:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280816896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828878.182236:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828878.182241:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828878.182243:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828878.182243:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828878/real 1787828878] req@ffff8ab3f075f840 x1874671280816640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828878.182250:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3427 00000100:00080000:0.0:1787828878.182254:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828878.182261:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828878.182266:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828878.182273:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280816640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828878.182277:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280817024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828878.182296:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828878.182297:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828878.182303:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828878.182304:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787828878.182305:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828878.182311:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280816640:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828878.182312:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3427 00000100:00080000:0.0:1787828878.182318:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828878.182321:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828878.182324:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828878.182331:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280817152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828878.182336:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280816768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828878.182359:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828878.182369:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828878/real 1787828878] req@ffff8ab3f075cb40 x1874671280816768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828878.182378:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280816768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828878.182384:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828878.182386:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828878.182388:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280816768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828878.182392:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280816896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828878.182401:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828878.182407:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828878/real 1787828878] req@ffff8ab3f075f0c0 x1874671280816896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828878.182414:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280816896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828878.182419:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828878.182421:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828878.182424:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280816896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828878.182428:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280817024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828878.182438:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828878.182443:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828878/real 1787828878] req@ffff8ab3f075de00 x1874671280817024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828878.182450:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280817024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828878.182457:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828878.182459:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828878.182461:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280817024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828878.182500:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280817152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828878.182522:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828878.182534:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828878/real 1787828878] req@ffff8ab3f075cf00 x1874671280817152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828878.182547:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280817152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828878.182558:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828878.182561:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828878.182565:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280817152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828879.206054:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828880.230061:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828881.254026:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828882.278017:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828883.301995:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828883.302003:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828883.302007:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828883.302016:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3432 00000100:00080000:3.0:1787828883.302025:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787828883.302026:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828883.302032:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828883.302072:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280817280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828883.302093:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828883.302095:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828883.302107:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828883.302109:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3432 00000100:00080000:3.0:1787828883.302112:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828883.302115:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828883.302123:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828883.302127:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280817280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787828883.302135:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280817408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828883.302144:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828883.302146:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828883.302147:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828883.302150:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3432 00000100:00080000:3.0:1787828883.302152:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828883.302155:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828883.302158:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828883.302177:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280817536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828883.302183:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828883.302185:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828883.302186:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828883.302189:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3432 00000100:00080000:3.0:1787828883.302191:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828883.302194:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828883.302197:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828883.302205:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280817664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828883.302211:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828883.302213:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828883.302215:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828883.302218:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3432 00000100:00080000:3.0:1787828883.302220:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828883.302224:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828883.302228:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828883.302237:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280817792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828883.302243:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280817408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828883.302276:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280817536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828883.302291:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280817664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828883.302305:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280817792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828884.326111:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828885.350046:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828886.374095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828886.374174:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787828886.374207:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828886.374220:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828886.374256:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828886.374267:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828886.374277:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787828886.374303:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:2.0:1787828886.374303:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828883/real 1787828886] req@ffff8ab3d0a421c0 x1874671280817408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828886.374342:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280817408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828886.374357:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828886.374363:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828886.374367:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280817408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828886.374376:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828883/real 1787828886] req@ffff8ab3d0a40000 x1874671280817536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828886.374389:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280817536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828886.374404:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828886.374424:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828886.374427:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280817536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828886.374434:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828883/real 1787828886] req@ffff8ab3d0a43840 x1874671280817664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828886.374460:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280817664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828886.374471:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828886.374475:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828886.374479:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280817664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828886.374486:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828883/real 1787828886] req@ffff8ab3d0a412c0 x1874671280817280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828886.374500:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280817280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828886.374509:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828886.374513:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828886.374528:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280817280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828886.374567:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828883/real 1787828886] req@ffff8ab3d0a43480 x1874671280817792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828886.374580:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280817792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828886.374590:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828886.374594:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828886.374598:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280817792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828887.398036:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828888.422085:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828888.422095:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828888.422105:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828888.422110:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828888.422123:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3438 00000100:00080000:2.0:1787828888.422134:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828888.422141:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828888.422195:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280817920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828888.422235:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828888.422243:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828888.422246:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828888.422253:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3438 00000100:00080000:2.0:1787828888.422262:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828888.422271:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828888.422286:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828888.422293:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280817920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828888.422315:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280818048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828888.422331:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828888.422336:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828888.422338:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828888.422344:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3438 00000100:00080000:2.0:1787828888.422349:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828888.422354:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828888.422361:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828888.422381:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280818176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828888.422418:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828888.422431:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828888.422447:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828888.422451:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828888.422460:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3438 00000100:00080000:2.0:1787828888.422469:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828888.422477:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828888.422488:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828888.422524:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828888/real 1787828888] req@ffff8ab3f043b840 x1874671280817920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828888.422527:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280818304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828888.422597:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828888.422602:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828888.422607:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280817920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828888.422609:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828888.422625:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3438 00000100:00080000:2.0:1787828888.422638:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828888.422641:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828888.422648:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787828888.422661:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000008:00100000:2.0:1787828888.422662:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828888.422701:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280817920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828888.422712:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280818432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828888.422744:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280818048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828888.422785:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828888.422807:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828888/real 1787828888] req@ffff8ab3f043b480 x1874671280818048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828888.422826:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280818048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828888.422839:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828888.422845:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828888.422851:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280818048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828888.422861:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280818176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828888.422927:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828888.422947:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828888/real 1787828888] req@ffff8ab3f0439e00 x1874671280818176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828888.422977:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280818176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828888.422989:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828888.422994:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828888.422999:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280818176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828888.423007:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280818304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828888.423037:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828888.423053:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828888/real 1787828888] req@ffff8ab3f043a1c0 x1874671280818304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828888.423070:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280818304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828888.423084:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828888.423089:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828888.423094:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280818304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828888.423102:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280818432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828888.423130:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828888.423146:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828888/real 1787828888] req@ffff8ab3f043a940 x1874671280818432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828888.423163:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280818432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828888.423179:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828888.423184:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828888.423189:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280818432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828889.446060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828890.470099:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828891.494122:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828892.518084:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828893.541999:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828893.542012:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828893.542018:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828893.542032:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3443 00010000:00010000:2.0:1787828893.542035:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828893.542047:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828893.542056:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828893.542101:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280818560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828893.542140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828893.542143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828893.542145:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828893.542148:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3443 00000100:00080000:3.0:1787828893.542150:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828893.542153:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828893.542163:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828893.542185:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280818688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828893.542193:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828893.542196:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828893.542197:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828893.542201:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3443 00000100:00080000:3.0:1787828893.542203:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828893.542208:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828893.542216:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280818560:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787828893.542216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828893.542237:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280818816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828893.542247:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828893.542250:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828893.542252:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828893.542255:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3443 00000100:00080000:3.0:1787828893.542257:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828893.542260:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828893.542264:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828893.542274:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280818944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828893.542281:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828893.542284:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828893.542286:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828893.542291:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3443 00000800:00000100:1.0:1787828893.542293:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828893.542296:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828893.542302:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828893.542306:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828893.542321:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280819072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828893.542382:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828893/real 1787828893] req@ffff8ab3d0a41e00 x1874671280818560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828893.542398:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280818560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828893.542407:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828893.542411:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828893.542414:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280818560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828893.542421:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280818688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828893.542439:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828893.542448:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828893/real 1787828893] req@ffff8ab3d0a430c0 x1874671280818688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828893.542456:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280818688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828893.542463:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828893.542465:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828893.542467:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280818688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828893.542490:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280818816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828893.542502:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828893.542508:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828893/real 1787828893] req@ffff8ab3d0a43c00 x1874671280818816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828893.542516:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280818816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828893.542538:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828893.542542:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828893.542546:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280818816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828893.542553:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280818944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828893.542575:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828893.542587:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828893/real 1787828893] req@ffff8ab3d0a43480 x1874671280818944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828893.542599:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280818944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828893.542609:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828893.542613:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828893.542617:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280818944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828893.542624:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280819072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828893.542645:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828893.542658:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828893/real 1787828893] req@ffff8ab3d0a412c0 x1874671280819072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828893.542671:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280819072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828893.542679:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828893.542681:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828893.542683:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280819072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828894.566035:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828895.590126:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828896.614105:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828897.638012:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828898.661997:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828898.662006:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1787828898.662009:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828898.662010:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828898.662021:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3448 00000100:00080000:2.0:1787828898.662029:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828898.662035:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828898.662068:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280819200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828898.662089:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828898.662092:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828898.662094:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828898.662097:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3448 00000100:00080000:2.0:1787828898.662099:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828898.662103:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828898.662111:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828898.662123:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280819328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828898.662125:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280819200:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828898.662144:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828898.662147:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828898.662148:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828898.662151:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3448 00000100:00080000:2.0:1787828898.662153:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828898.662156:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828898.662160:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828898.662168:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280819456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828898.662175:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828898.662178:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828898.662179:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828898.662181:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3448 00000800:00000100:1.0:1787828898.662185:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828898.662185:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828898.662191:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828898.662195:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828898.662203:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280819584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828898.662210:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828898.662213:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828898.662214:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828898.662217:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3448 00000100:00080000:2.0:1787828898.662221:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828898.662225:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828898.662229:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828898.662232:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828898/real 1787828898] req@ffff8ab3f0439a40 x1874671280819200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828898.662239:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280819712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828898.662253:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280819200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828898.662263:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828898.662266:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828898.662269:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280819200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828898.662287:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280819328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828898.662301:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828898.662309:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828898/real 1787828898] req@ffff8ab3f043ad00 x1874671280819328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828898.662317:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280819328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828898.662343:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828898.662346:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828898.662348:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280819328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828898.662353:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280819456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828898.662364:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828898.662371:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828898/real 1787828898] req@ffff8ab3f04392c0 x1874671280819456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828898.662378:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280819456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828898.662383:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828898.662386:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828898.662387:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280819456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828898.662391:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280819584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828898.662400:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828898.662405:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828898/real 1787828898] req@ffff8ab3f04383c0 x1874671280819584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828898.662412:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280819584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828898.662418:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828898.662420:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828898.662422:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280819584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828898.662426:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280819712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828898.662436:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828898.662441:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828898/real 1787828898] req@ffff8ab3f043bc00 x1874671280819712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828898.662448:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280819712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828898.662454:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828898.662456:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828898.662457:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280819712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828899.686035:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828900.710055:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828901.734072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828902.758029:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828903.781972:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828903.781981:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828903.781985:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828903.781993:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3453 00000100:00080000:2.0:1787828903.782001:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828903.782015:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828903.782042:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828903.782054:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280819840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828903.782074:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828903.782076:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828903.782078:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828903.782080:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3453 00000100:00080000:2.0:1787828903.782083:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828903.782087:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828903.782095:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828903.782105:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280819968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828903.782116:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280819840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828903.782121:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828903.782126:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828903.782128:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828903.782132:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3453 00000100:00080000:2.0:1787828903.782134:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828903.782137:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828903.782140:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828903.782150:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280820096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828903.782156:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828903.782159:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828903.782161:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828903.782164:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3453 00000100:00080000:2.0:1787828903.782173:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828903.782176:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828903.782180:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828903.782184:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828903.782191:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280820224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828903.782200:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828903.782203:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828903.782204:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828903.782207:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3453 00000100:00080000:2.0:1787828903.782210:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828903.782212:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828903.782215:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828903.782222:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280820352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828903.782239:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828903/real 1787828903] req@ffff8ab3f0439680 x1874671280819840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828903.782260:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280819840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828903.782272:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828903.782277:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828903.782282:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280819840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828903.782321:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280819968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828903.782346:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828903.782361:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828903/real 1787828903] req@ffff8ab3f043a580 x1874671280819968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828903.782376:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280819968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828903.782385:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828903.782389:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828903.782393:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280819968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828903.782400:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280820096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828903.782420:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828903.782433:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828903/real 1787828903] req@ffff8ab3f0438f00 x1874671280820096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828903.782446:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280820096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828903.782455:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828903.782459:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828903.782463:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280820096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828903.782470:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280820224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828903.782490:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828903.782502:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828903/real 1787828903] req@ffff8ab3f0438b40 x1874671280820224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828903.782514:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280820224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828903.782525:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828903.782529:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828903.782532:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280820224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828903.782539:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280820352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828903.782558:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828903.782569:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828903/real 1787828903] req@ffff8ab3f0438780 x1874671280820352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828903.782591:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280820352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828903.782601:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828903.782605:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828903.782608:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280820352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828904.806228:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828905.830175:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828906.854086:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828907.878077:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828908.902066:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828908.902078:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828908.902082:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828908.902093:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3458 00000100:00080000:0.0:1787828908.902107:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828908.902116:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828908.902122:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828908.902155:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280820480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828908.902185:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828908.902189:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828908.902192:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828908.902197:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3458 00000100:00080000:0.0:1787828908.902203:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828908.902208:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828908.902219:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828908.902231:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280820480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787828908.902241:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280820608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828908.902270:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828908.902274:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828908.902277:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828908.902282:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3458 00000100:00080000:0.0:1787828908.902287:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828908.902292:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828908.902298:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828908.902313:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280820736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828908.902325:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828908.902329:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828908.902331:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828908.902336:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3458 00000100:00080000:0.0:1787828908.902341:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828908.902345:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828908.902352:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828908.902361:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828908.902378:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280820864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828908.902397:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828908.902401:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828908.902403:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828908.902408:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3458 00000100:00100000:1.0:1787828908.902415:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828908/real 1787828908] req@ffff8ab3f075c000 x1874671280820480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828908.902416:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828908.902436:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828908.902446:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280820480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828908.902447:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787828908.902467:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828908.902474:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787828908.902475:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280820992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828908.902485:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280820480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828908.902523:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280820608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828908.902542:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828908.902552:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828908/real 1787828908] req@ffff8ab3f075cf00 x1874671280820608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828908.902560:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280820608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828908.902567:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828908.902569:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828908.902572:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280820608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828908.902576:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280820736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828908.902586:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828908.902592:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828908/real 1787828908] req@ffff8ab3f075de00 x1874671280820736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828908.902599:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280820736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828908.902605:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828908.902607:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828908.902608:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280820736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828908.902612:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280820864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828908.902622:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828908.902628:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828908/real 1787828908] req@ffff8ab3f075f0c0 x1874671280820864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828908.902636:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280820864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828908.902643:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828908.902645:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828908.902646:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280820864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828908.902650:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280820992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828908.902660:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828908.902666:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828908/real 1787828908] req@ffff8ab3f075cb40 x1874671280820992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828908.902673:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280820992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828908.902682:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828908.902685:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828908.902688:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280820992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828909.926104:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828910.950058:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828911.974049:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828912.998040:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828914.022016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828914.022024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828914.022029:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828914.022039:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3463 00000100:00080000:3.0:1787828914.022048:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828914.022054:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828914.022088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280821120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828914.022097:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828914.022124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828914.022128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828914.022129:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828914.022132:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3463 00000100:00080000:3.0:1787828914.022135:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828914.022138:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828914.022147:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828914.022155:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280821248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828914.022164:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828914.022167:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828914.022169:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280821120:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828914.022169:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828914.022186:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3463 00000100:00080000:3.0:1787828914.022189:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828914.022200:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828914.022208:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828914.022234:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280821376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828914.022241:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828914.022244:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828914.022245:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828914.022249:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3463 00000100:00080000:3.0:1787828914.022252:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828914.022256:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828914.022271:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828914.022282:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280821504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828914.022289:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828914.022292:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828914.022293:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828914.022296:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3463 00000100:00080000:3.0:1787828914.022307:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828914.022310:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828914.022313:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787828914.022321:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787828914.022322:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280821632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828914.022390:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828914/real 1787828914] req@ffff8ab3d0a43840 x1874671280821120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828914.022407:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280821120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828914.022419:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828914.022422:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828914.022425:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280821120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828914.022452:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280821248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828914.022472:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828914.022486:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828914/real 1787828914] req@ffff8ab3d0a40000 x1874671280821248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828914.022500:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280821248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828914.022510:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828914.022514:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828914.022518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280821248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828914.022525:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280821376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828914.022542:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828914.022550:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828914/real 1787828914] req@ffff8ab3d0a421c0 x1874671280821376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828914.022558:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280821376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828914.022565:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828914.022567:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828914.022570:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280821376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828914.022575:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280821504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828914.022587:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828914.022592:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828914/real 1787828914] req@ffff8ab3d0a412c0 x1874671280821504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828914.022599:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280821504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828914.022607:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828914.022608:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828914.022611:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280821504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828914.022617:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280821632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828914.022636:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828914.022647:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828914/real 1787828914] req@ffff8ab3d0a43480 x1874671280821632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828914.022661:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280821632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828914.022670:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828914.022673:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828914.022675:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280821632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828915.046124:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828916.070015:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828917.094087:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828918.118025:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828919.141976:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828919.141984:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828919.141988:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828919.141996:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3468 00000100:00080000:3.0:1787828919.142005:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828919.142012:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787828919.142025:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828919.142047:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280821760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828919.142083:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828919.142085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828919.142086:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828919.142089:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3468 00000100:00080000:3.0:1787828919.142091:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828919.142094:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828919.142103:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828919.142112:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280821888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828919.142118:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280821760:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828919.142122:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828919.142126:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828919.142127:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828919.142131:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3468 00000100:00080000:3.0:1787828919.142133:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828919.142135:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828919.142140:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828919.142153:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280822016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828919.142164:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828919.142168:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828919.142170:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828919.142174:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787828919.142191:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3468 00000100:00080000:3.0:1787828919.142198:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828919.142203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828919.142209:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787828919.142221:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280822144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828919.142229:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828919.142231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828919.142235:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828919.142236:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828919/real 1787828919] req@ffff8ab3d0a43c00 x1874671280821760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828919.142241:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3468 00000100:00080000:3.0:1787828919.142246:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828919.142252:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828919.142256:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828919.142262:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280821760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828919.142266:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280822272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828919.142284:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828919.142289:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828919.142293:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280821760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828919.142314:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280821888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828919.142330:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828919.142339:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828919/real 1787828919] req@ffff8ab3d0a430c0 x1874671280821888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828919.142347:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280821888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828919.142353:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828919.142355:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828919.142357:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280821888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828919.142361:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280822016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828919.142370:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828919.142376:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828919/real 1787828919] req@ffff8ab3d0a41e00 x1874671280822016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828919.142383:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280822016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828919.142392:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828919.142394:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828919.142396:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280822016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828919.142399:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280822144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828919.142409:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828919.142414:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828919/real 1787828919] req@ffff8ab3d0a43480 x1874671280822144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828919.142421:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280822144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828919.142427:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828919.142429:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828919.142431:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280822144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828919.142434:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280822272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828919.142444:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828919.142454:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828919/real 1787828919] req@ffff8ab3d0a412c0 x1874671280822272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828919.142466:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280822272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828919.142476:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828919.142479:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828919.142483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280822272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828920.166043:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828921.190044:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828922.214036:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828923.238178:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828924.262120:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828924.262122:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828924.262135:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828924.262184:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828924.262198:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3473 00000100:00080000:2.0:1787828924.262210:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828924.262219:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828924.262288:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280822400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828924.262326:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828924.262331:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828924.262334:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828924.262338:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3473 00000100:00080000:2.0:1787828924.262344:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828924.262352:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828924.262366:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828924.262385:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280822528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828924.262390:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280822400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828924.262415:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828924.262420:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828924.262422:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828924.262428:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3473 00000100:00080000:2.0:1787828924.262450:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828924.262456:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828924.262462:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828924.262477:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280822656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828924.262490:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828924.262497:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828924.262501:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828924.262509:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828924.262533:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3473 00000100:00080000:2.0:1787828924.262545:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828924.262556:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828924.262568:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828924.262593:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280822784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828924.262613:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828924.262622:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828924.262625:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828924/real 1787828924] req@ffff8ab3f043b0c0 x1874671280822400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828924.262628:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828924.262647:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3473 00000100:00080000:2.0:1787828924.262662:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828924.262673:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280822400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828924.262676:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828924.262697:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787828924.262701:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828924.262712:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828924.262719:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280822400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828924.262726:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280822912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828924.262759:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280822528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828924.262803:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828924.262830:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828924/real 1787828924] req@ffff8ab3f0438780 x1874671280822528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828924.262857:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280822528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828924.262875:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828924.262882:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828924.262924:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280822528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828924.262934:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280822656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828924.262981:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828924.262996:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828924/real 1787828924] req@ffff8ab3f0438b40 x1874671280822656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828924.263010:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280822656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828924.263021:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828924.263024:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828924.263029:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280822656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828924.263054:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280822784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828924.263077:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828924.263088:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828924/real 1787828924] req@ffff8ab3f0438f00 x1874671280822784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828924.263102:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280822784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828924.263114:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828924.263117:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828924.263121:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280822784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828924.263127:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280822912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828924.263177:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828924.263193:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828924/real 1787828924] req@ffff8ab3f043a580 x1874671280822912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828924.263208:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280822912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828924.263220:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828924.263224:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828924.263228:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280822912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787828925.286088:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828926.310042:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828927.334020:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828928.358068:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828929.382038:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828929.382047:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828929.382051:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828929.382060:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3479 00000100:00080000:2.0:1787828929.382072:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828929.382081:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787828929.382129:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828929.382131:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280823040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828929.382171:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828929.382174:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828929.382175:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828929.382179:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3479 00000100:00080000:2.0:1787828929.382182:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828929.382185:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828929.382195:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828929.382206:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280823168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828929.382217:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828929.382225:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828929.382226:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280823040:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828929.382228:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828929.382246:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3479 00000100:00080000:2.0:1787828929.382250:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828929.382252:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828929.382256:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828929.382279:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280823296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828929.382286:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828929.382289:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828929.382290:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828929.382293:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3479 00000100:00080000:2.0:1787828929.382295:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828929.382298:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828929.382301:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828929.382310:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280823424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828929.382317:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828929.382319:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828929.382321:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828929.382324:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3479 00000100:00080000:2.0:1787828929.382327:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828929.382340:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828929.382343:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828929.382351:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280823552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828929.382361:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828929.382455:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828929/real 1787828929] req@ffff8ab3f0439680 x1874671280823040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828929.382474:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280823040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828929.382485:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828929.382489:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828929.382509:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280823040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828929.382535:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280823168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828929.382553:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828929.382580:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828929/real 1787828929] req@ffff8ab3f043bc00 x1874671280823168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828929.382594:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280823168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828929.382606:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828929.382610:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828929.382624:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280823168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828929.382635:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280823296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828929.382674:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828929.382686:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828929/real 1787828929] req@ffff8ab3f04383c0 x1874671280823296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828929.382699:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280823296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828929.382707:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828929.382710:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828929.382730:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280823296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828929.382738:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280823424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828929.382760:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828929.382770:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828929/real 1787828929] req@ffff8ab3f04392c0 x1874671280823424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828929.382780:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280823424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828929.382789:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828929.382792:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828929.382794:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280823424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828929.382799:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280823552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828929.382813:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828929.382819:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828929/real 1787828929] req@ffff8ab3f043ad00 x1874671280823552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828929.382829:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280823552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828929.382837:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828929.382839:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828929.382841:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280823552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828930.406130:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828931.430059:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828932.454083:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828933.478046:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828934.501968:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828934.501977:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828934.501981:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828934.501990:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3484 00000100:00080000:2.0:1787828934.501999:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828934.502005:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828934.502041:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828934.502042:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280823680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828934.502085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828934.502090:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828934.502092:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828934.502098:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3484 00000100:00080000:2.0:1787828934.502103:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828934.502108:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828934.502117:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280823680:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787828934.502119:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828934.502138:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280823808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828934.502156:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828934.502158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828934.502160:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828934.502163:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3484 00000100:00080000:2.0:1787828934.502167:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828934.502172:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828934.502172:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828934.502180:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828934.502193:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280823936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828934.502199:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828934.502202:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828934.502204:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828934.502207:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3484 00000100:00080000:2.0:1787828934.502210:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828934.502214:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828934.502219:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828934/real 1787828934] req@ffff8ab3f0439a40 x1874671280823680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828934.502219:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828934.502233:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280824064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828934.502240:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280823680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828934.502243:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828934.502251:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828934.502255:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828934.502263:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828934.502263:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3484 00000100:00080000:2.0:1787828934.502270:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828934.502272:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787828934.502276:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828934.502278:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280823680:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787828934.502280:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828934.502297:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280824192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828934.502301:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280823808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828934.502326:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828934.502344:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828934/real 1787828934] req@ffff8ab3f043a940 x1874671280823808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828934.502353:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280823808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828934.502363:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828934.502365:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828934.502367:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280823808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828934.502371:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280823936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828934.502382:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828934.502388:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828934/real 1787828934] req@ffff8ab3f043a1c0 x1874671280823936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828934.502395:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280823936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828934.502400:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828934.502402:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828934.502404:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280823936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828934.502407:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280824064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828934.502417:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828934.502422:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828934/real 1787828934] req@ffff8ab3f0439e00 x1874671280824064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828934.502429:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280824064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828934.502436:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828934.502437:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828934.502440:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280824064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828934.502449:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280824192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828934.502458:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828934.502464:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828934/real 1787828934] req@ffff8ab3f043b480 x1874671280824192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828934.502470:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280824192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828934.502476:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828934.502478:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828934.502480:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280824192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787828935.526048:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828936.550081:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828937.574017:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828938.598108:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828939.621954:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828939.621963:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828939.621978:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828939.621987:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3489 00000100:00080000:2.0:1787828939.621996:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828939.622002:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828939.622053:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280824320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787828939.622062:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828939.622091:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828939.622095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828939.622097:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828939.622102:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3489 00000100:00080000:2.0:1787828939.622106:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828939.622109:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828939.622117:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828939.622130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280824448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828939.622149:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828939.622156:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280824320:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828939.622156:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828939.622160:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828939.622168:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3489 00000100:00080000:2.0:1787828939.622170:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828939.622173:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828939.622177:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828939.622186:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280824576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828939.622193:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828939.622196:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828939.622198:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828939.622200:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3489 00000100:00080000:2.0:1787828939.622210:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828939.622213:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828939.622216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828939.622220:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828939.622228:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280824704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828939.622236:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828939.622238:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828939.622240:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828939.622243:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3489 00000100:00080000:2.0:1787828939.622245:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828939.622248:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828939.622258:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828939.622262:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828939/real 1787828939] req@ffff8ab3f043b840 x1874671280824320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828939.622270:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280824832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828939.622282:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280824320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828939.622292:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828939.622295:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828939.622298:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280824320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828939.622315:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280824448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828939.622329:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828939.622337:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828939/real 1787828939] req@ffff8ab3f043b480 x1874671280824448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828939.622344:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280824448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828939.622350:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828939.622352:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828939.622355:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280824448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828939.622358:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280824576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828939.622367:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828939.622373:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828939/real 1787828939] req@ffff8ab3f0439e00 x1874671280824576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828939.622380:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280824576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828939.622385:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828939.622387:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828939.622389:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280824576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828939.622392:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280824704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828939.622401:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828939.622407:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828939/real 1787828939] req@ffff8ab3f043a1c0 x1874671280824704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828939.622413:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280824704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828939.622420:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828939.622422:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828939.622424:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280824704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828939.622427:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280824832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828939.622436:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828939.622441:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828939/real 1787828939] req@ffff8ab3f043a940 x1874671280824832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828939.622447:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280824832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828939.622454:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828939.622455:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828939.622457:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280824832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828940.646009:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828941.670095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828942.694061:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828943.718038:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828944.741989:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828944.742012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828944.742021:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828944.742026:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828944.742036:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3494 00000100:00080000:3.0:1787828944.742047:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828944.742054:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787828944.742091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280824960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828944.742118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828944.742122:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828944.742124:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828944.742130:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3494 00000100:00080000:3.0:1787828944.742134:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828944.742140:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828944.742147:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280824960:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787828944.742150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828944.742170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280825088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828944.742183:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828944.742187:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828944.742189:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828944.742193:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3494 00000100:00080000:3.0:1787828944.742198:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828944.742204:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828944.742210:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787828944.742210:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828944.742227:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280825216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828944.742245:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828944.742249:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828944.742252:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828944.742256:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828944/real 1787828944] req@ffff8ab3d0a421c0 x1874671280824960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828944.742260:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3494 00000100:00080000:3.0:1787828944.742267:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828944.742276:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828944.742282:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828944.742288:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280824960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828944.742299:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280825344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828944.742303:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828944.742314:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787828944.742320:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828944.742321:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280824960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787828944.742329:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828944.742333:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828944.742339:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3494 00000100:00080000:3.0:1787828944.742343:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828944.742349:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280825088:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787828944.742350:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787828944.742360:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787828944.742376:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280825472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828944.742378:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828944.742392:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828944/real 1787828944] req@ffff8ab3d0a40000 x1874671280825088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828944.742402:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280825088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828944.742409:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828944.742412:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828944.742414:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280825088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828944.742419:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280825216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828944.742430:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828944.742435:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828944/real 1787828944] req@ffff8ab3d0a43840 x1874671280825216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828944.742442:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280825216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828944.742448:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828944.742449:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828944.742451:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280825216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828944.742455:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280825344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828944.742465:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828944.742471:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828944/real 1787828944] req@ffff8ab3d0a412c0 x1874671280825344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828944.742478:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280825344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828944.742485:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828944.742487:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828944.742489:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280825344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828944.742500:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280825472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828944.742518:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828944.742525:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828944/real 1787828944] req@ffff8ab3d0a43480 x1874671280825472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828944.742533:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280825472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828944.742540:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828944.742542:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828944.742544:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280825472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787828945.766017:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828946.790023:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828947.814090:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828948.838069:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828949.861979:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828949.861987:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828949.861991:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828949.862010:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3499 00000100:00080000:1.0:1787828949.862019:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828949.862025:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828949.862059:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280825600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828949.862092:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828949.862097:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828949.862099:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828949.862103:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3499 00000100:00080000:1.0:1787828949.862108:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828949.862113:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828949.862123:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828949.862134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280825728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828949.862144:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280825600:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787828949.862145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828949.862153:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828949.862156:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828949.862160:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3499 00000100:00080000:1.0:1787828949.862162:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828949.862166:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828949.862169:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828949.862179:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280825856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828949.862186:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828949.862189:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828949.862190:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828949.862193:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3499 00000100:00080000:1.0:1787828949.862197:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828949.862200:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828949.862204:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828949.862211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280825984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828949.862218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828949.862221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828949.862222:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828949.862224:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3499 00000100:00080000:1.0:1787828949.862226:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828949.862228:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828949.862232:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828949.862241:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280826112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828949.862256:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280825728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828949.862294:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280825856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828949.862310:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280825984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828949.862327:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280826112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787828949.862341:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828950.886065:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828951.910056:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787828952.934108:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828952.934227:0:10942:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:3.0:1787828952.934271:0:10942:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828952.934288:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787828952.934337:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787828952.934351:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787828952.934362:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787828952.934374:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:2.0:1787828952.934385:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828949/real 1787828952] req@ffff8ab3d22fd2c0 x1874671280825728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828952.934414:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280825728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828952.934428:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828952.934434:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828952.934440:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280825728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828952.934449:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828949/real 1787828952] req@ffff8ab3d22fde00 x1874671280825856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828952.934466:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280825856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828952.934474:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828952.934478:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828952.934482:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280825856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828952.934488:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828949/real 1787828952] req@ffff8ab3d22fe1c0 x1874671280825984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828952.934499:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280825984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828952.934509:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828952.934512:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828952.934515:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280825984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828952.934523:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828949/real 1787828952] req@ffff8ab3d22fc780 x1874671280826112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828952.934553:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280826112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828952.934563:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828952.934566:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828952.934569:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280826112:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828952.934575:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828949/real 1787828952] req@ffff8ab3d22fed00 x1874671280825600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828952.934587:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280825600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828952.934595:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828952.934598:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828952.934602:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280825600:192.168.203.124@tcp:250:kworker.0 00010000:00010000:0.0:1787828953.958062:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828954.982059:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1787828954.982067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828954.982070:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828954.982077:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828954.982091:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3504 00000100:00080000:2.0:1787828954.982103:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828954.982113:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787828954.982155:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280826240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828954.982186:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828954.982188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828954.982190:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828954.982193:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3504 00000100:00080000:2.0:1787828954.982196:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828954.982199:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828954.982210:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828954.982224:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280826368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828954.982248:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828954.982254:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280826240:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828954.982254:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828954.982257:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828954.982267:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3504 00000100:00080000:2.0:1787828954.982272:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828954.982277:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828954.982283:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828954.982298:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280826496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828954.982308:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828954.982312:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828954.982314:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828954.982319:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3504 00000100:00080000:2.0:1787828954.982332:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828954.982337:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828954.982343:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787828954.982346:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828954.982359:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280826624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828954.982370:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828954.982374:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828954.982375:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828954.982380:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3504 00000100:00080000:2.0:1787828954.982385:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828954.982390:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828954.982395:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828954.982410:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280826752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828954.982413:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828954/real 1787828954] req@ffff8ab3f0439a40 x1874671280826240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828954.982452:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280826240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828954.982466:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828954.982471:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828954.982477:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280826240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828954.982510:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280826368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828954.982538:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828954.982555:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828954/real 1787828954] req@ffff8ab3f043ad00 x1874671280826368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828954.982569:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280826368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828954.982578:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828954.982582:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828954.982586:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280826368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828954.982593:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280826496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828954.982616:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828954.982628:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828954/real 1787828954] req@ffff8ab3f04392c0 x1874671280826496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828954.982640:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280826496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828954.982664:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828954.982668:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828954.982672:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280826496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828954.982678:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280826624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828954.982701:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828954.982715:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828954/real 1787828954] req@ffff8ab3f04383c0 x1874671280826624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828954.982728:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280826624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828954.982739:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828954.982756:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828954.982760:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280826624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828954.982767:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280826752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828954.982788:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828954.982800:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828954/real 1787828954] req@ffff8ab3f043bc00 x1874671280826752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828954.982812:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280826752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828954.982823:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828954.982827:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828954.982830:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280826752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787828956.006128:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828957.030102:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828958.054129:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828959.078065:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828960.101991:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828960.102000:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828960.102004:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828960.102014:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3509 00000100:00080000:2.0:1787828960.102022:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828960.102029:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787828960.102063:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828960.102063:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280826880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828960.102089:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828960.102091:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828960.102092:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828960.102095:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3509 00000100:00080000:2.0:1787828960.102097:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828960.102100:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828960.102108:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828960.102119:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280827008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828960.102127:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828960.102132:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828960.102134:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828960.102143:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280826880:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828960.102143:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3509 00000100:00080000:2.0:1787828960.102152:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828960.102160:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828960.102163:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828960.102176:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280827136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828960.102183:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828960.102186:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828960.102188:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828960.102192:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3509 00000100:00080000:2.0:1787828960.102194:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828960.102198:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828960.102202:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828960.102209:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280827264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828960.102215:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828960.102218:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828960.102219:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828960.102223:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3509 00000100:00080000:2.0:1787828960.102225:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828960.102227:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828960.102231:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828960.102241:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280827392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828960.102244:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828960.102316:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828960/real 1787828960] req@ffff8ab3f0439680 x1874671280826880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828960.102335:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280826880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828960.102347:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828960.102351:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828960.102357:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280826880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828960.102390:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280827008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828960.102419:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828960.102432:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828960/real 1787828960] req@ffff8ab3f043a580 x1874671280827008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828960.102443:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280827008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828960.102451:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828960.102454:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828960.102456:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280827008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828960.102462:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280827136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828960.102477:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828960.102485:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828960/real 1787828960] req@ffff8ab3f0438f00 x1874671280827136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828960.102494:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280827136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828960.102500:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828960.102502:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828960.102506:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280827136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828960.102511:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280827264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828960.102524:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828960.102531:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828960/real 1787828960] req@ffff8ab3f0438b40 x1874671280827264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828960.102539:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280827264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828960.102547:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828960.102550:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828960.102552:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280827264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828960.102556:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280827392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828960.102567:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828960.102574:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828960/real 1787828960] req@ffff8ab3f0438780 x1874671280827392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828960.102583:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280827392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828960.102591:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828960.102593:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828960.102595:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280827392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828961.126085:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828962.150084:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828963.174104:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828964.198019:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828965.222022:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828965.222032:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828965.222037:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828965.222047:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3514 00000100:00080000:2.0:1787828965.222059:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828965.222067:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828965.222102:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828965.222120:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280827520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828965.222148:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828965.222152:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828965.222164:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828965.222169:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3514 00000100:00080000:2.0:1787828965.222174:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828965.222179:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828965.222189:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828965.222200:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280827520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828965.222207:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280827648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828965.222234:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828965.222238:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828965.222241:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828965.222246:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3514 00000100:00080000:2.0:1787828965.222270:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828965.222277:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828965.222282:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787828965.222298:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828965.222316:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280827776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828965.222327:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828965.222331:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828965.222333:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828965.222338:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3514 00000100:00080000:2.0:1787828965.222343:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828965.222348:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828965/real 1787828965] req@ffff8ab3f043b0c0 x1874671280827520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828965.222352:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828965.222361:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828965.222376:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280827520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828965.222379:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280827904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828965.222403:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828965.222404:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828965.222412:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828965.222415:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787828965.222416:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828965.222424:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280827520:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828965.222425:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3514 00000100:00080000:2.0:1787828965.222432:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828965.222438:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828965.222443:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828965.222455:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280828032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828965.222458:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280827648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828965.222494:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828965.222511:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828965/real 1787828965] req@ffff8ab3f0438780 x1874671280827648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828965.222524:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280827648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828965.222532:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828965.222536:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828965.222539:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280827648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828965.222545:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280827776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828965.222575:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828965.222588:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828965/real 1787828965] req@ffff8ab3f0438b40 x1874671280827776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828965.222601:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280827776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828965.222615:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828965.222618:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828965.222622:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280827776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828965.222628:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280827904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828965.222649:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828965.222661:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828965/real 1787828965] req@ffff8ab3f0438f00 x1874671280827904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828965.222673:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280827904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828965.222693:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828965.222697:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828965.222701:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280827904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828965.222717:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280828032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828965.222739:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828965.222761:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828965/real 1787828965] req@ffff8ab3f043a580 x1874671280828032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828965.222773:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280828032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828965.222783:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828965.222786:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828965.222791:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280828032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787828966.246039:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828967.270033:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828968.294018:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828969.317992:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828970.341986:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828970.341996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828970.342000:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828970.342008:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3519 00000100:00080000:2.0:1787828970.342017:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828970.342023:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787828970.342052:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828970.342058:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280828160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828970.342079:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828970.342082:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828970.342084:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828970.342086:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3519 00000100:00080000:2.0:1787828970.342089:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828970.342093:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828970.342101:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828970.342114:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280828288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828970.342115:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280828160:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828970.342130:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828970.342133:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828970.342134:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828970.342137:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3519 00000100:00080000:2.0:1787828970.342155:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828970.342158:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828970.342163:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828970.342174:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280828416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828970.342181:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828970.342184:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828970.342185:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828970.342187:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3519 00000100:00080000:2.0:1787828970.342190:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828970.342194:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787828970.342197:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828970.342202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828970.342211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280828544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828970.342218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828970.342220:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828970.342222:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828970.342225:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3519 00000100:00080000:2.0:1787828970.342227:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828970.342230:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828970.342234:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828970.342238:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828970/real 1787828970] req@ffff8ab3f0439680 x1874671280828160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828970.342245:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280828672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828970.342259:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280828160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828970.342268:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828970.342271:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828970.342274:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280828160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828970.342291:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280828288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828970.342306:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828970.342315:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828970/real 1787828970] req@ffff8ab3f043bc00 x1874671280828288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828970.342323:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280828288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828970.342329:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828970.342331:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828970.342333:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280828288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828970.342336:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280828416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828970.342347:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828970.342353:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828970/real 1787828970] req@ffff8ab3f04383c0 x1874671280828416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828970.342360:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280828416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828970.342365:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828970.342367:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828970.342369:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280828416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828970.342373:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280828544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828970.342382:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828970.342388:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828970/real 1787828970] req@ffff8ab3f04392c0 x1874671280828544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828970.342395:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280828544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828970.342401:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828970.342403:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828970.342405:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280828544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828970.342408:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280828672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828970.342417:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828970.342422:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828970/real 1787828970] req@ffff8ab3f043ad00 x1874671280828672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828970.342429:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280828672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828970.342436:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828970.342438:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828970.342440:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280828672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828971.366051:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828972.390136:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828973.414095:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828974.438124:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828975.461985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828975.461996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828975.462002:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828975.462012:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3525 00000100:00080000:2.0:1787828975.462023:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828975.462032:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787828975.462042:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828975.462076:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280828800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828975.462100:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828975.462104:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828975.462106:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828975.462111:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3525 00000100:00080000:2.0:1787828975.462116:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828975.462121:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828975.462131:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828975.462135:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280828800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828975.462147:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280828928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828975.462161:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828975.462165:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828975.462167:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828975.462172:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3525 00000100:00080000:2.0:1787828975.462189:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828975.462195:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787828975.462199:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787828975.462204:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828975.462220:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280829056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828975.462239:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828975.462240:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828975/real 1787828975] req@ffff8ab3f0439a40 x1874671280828800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828975.462248:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828975.462252:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828975.462262:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3525 00000100:00080000:2.0:1787828975.462268:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828975.462272:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828975.462277:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280828800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828975.462278:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787828975.462291:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280829184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828975.462294:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828975.462302:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828975.462305:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787828975.462307:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828975.462311:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828975.462313:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280828800:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787828975.462317:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3525 00000100:00080000:2.0:1787828975.462323:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828975.462326:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787828975.462329:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787828975.462336:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280829312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828975.462343:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280828928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828975.462370:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828975.462381:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828975/real 1787828975] req@ffff8ab3f043a940 x1874671280828928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828975.462390:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280828928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828975.462395:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828975.462398:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828975.462400:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280828928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828975.462405:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280829056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787828975.462415:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828975.462424:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828975/real 1787828975] req@ffff8ab3f043a1c0 x1874671280829056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828975.462436:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280829056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828975.462445:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828975.462448:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828975.462452:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280829056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787828975.462456:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280829184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828975.462471:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828975.462478:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828975/real 1787828975] req@ffff8ab3f0439e00 x1874671280829184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828975.462486:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280829184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828975.462492:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828975.462494:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828975.462497:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280829184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787828975.462507:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280829312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787828975.462517:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828975.462522:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828975/real 1787828975] req@ffff8ab3f043b480 x1874671280829312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828975.462539:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280829312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828975.462546:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828975.462548:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787828975.462550:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280829312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787828976.486037:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828977.510056:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828978.534020:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828979.558037:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828980.581966:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828980.581974:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828980.581977:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828980.581985:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3530 00000100:00080000:1.0:1787828980.581994:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828980.582000:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828980.582031:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280829440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828980.582055:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828980.582059:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828980.582061:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828980.582065:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3530 00000100:00080000:1.0:1787828980.582069:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828980.582075:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828980.582085:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828980.582095:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280829440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787828980.582101:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280829568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828980.582117:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828980.582119:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828980.582121:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828980.582123:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3530 00000100:00080000:1.0:1787828980.582127:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828980.582129:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828980.582133:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828980.582142:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280829696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828980.582149:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828980.582151:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828980.582152:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828980.582155:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3530 00000100:00080000:1.0:1787828980.582157:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828980.582160:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787828980.582165:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:1.0:1787828980.582166:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828980.582177:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280829824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828980.582184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828980.582186:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828980.582188:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828980.582191:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3530 00000100:00080000:1.0:1787828980.582194:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828980.582196:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828980.582199:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828980.582208:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280829952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828980.582223:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828980/real 1787828980] req@ffff8ab3d22fd680 x1874671280829440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828980.582253:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280829440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828980.582265:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828980.582270:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828980.582275:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280829440:192.168.203.124@tcp:250:kworker.0 00010000:00010000:0.0:1787828980.582285:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828980.582302:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280829568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828980.582327:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828980.582340:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828980/real 1787828980] req@ffff8ab3d22fcb40 x1874671280829568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828980.582354:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280829568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828980.582364:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828980.582367:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828980.582369:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280829568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828980.582374:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280829696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828980.582394:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828980.582406:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828980/real 1787828980] req@ffff8ab3d22fda40 x1874671280829696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828980.582420:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280829696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828980.582436:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828980.582440:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828980.582444:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280829696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828980.582452:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280829824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828980.582474:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828980.582486:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828980/real 1787828980] req@ffff8ab3d22fc000 x1874671280829824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828980.582498:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280829824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828980.582509:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828980.582513:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828980.582517:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280829824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828980.582524:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280829952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828980.582543:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828980.582555:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828980/real 1787828980] req@ffff8ab3d22ff480 x1874671280829952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828980.582566:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280829952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828980.582577:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828980.582580:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828980.582584:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280829952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828981.606020:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828982.630053:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828983.654011:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828984.678051:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828985.701979:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828985.701989:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828985.701994:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828985.702004:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3535 00000100:00080000:1.0:1787828985.702034:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828985.702041:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828985.702090:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280830080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828985.702121:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828985.702126:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828985.702127:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828985.702131:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3535 00000100:00080000:1.0:1787828985.702136:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828985.702141:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828985.702152:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828985.702176:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280830208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828985.702189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828985.702192:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280830080:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787828985.702198:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828985.702200:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828985.702205:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3535 00000100:00080000:1.0:1787828985.702213:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828985.702220:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828985.702227:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828985.702241:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280830336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828985.702252:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828985.702256:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828985.702258:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828985.702263:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3535 00000100:00080000:1.0:1787828985.702267:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828985.702271:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828985.702287:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787828985.702287:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828985.702303:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280830464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828985.702313:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828985.702318:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828985.702320:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828985.702325:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3535 00000100:00080000:1.0:1787828985.702329:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828985.702333:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828985.702339:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787828985.702364:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280830592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828985.702368:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828985/real 1787828985] req@ffff8ab3d22ff480 x1874671280830080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828985.702397:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280830080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828985.702409:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828985.702415:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828985.702421:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280830080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828985.702447:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280830208:192.168.203.124@tcp:38:kworker.0 00010000:00010000:3.0:1787828985.702452:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1787828985.702476:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828985.702492:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828985/real 1787828985] req@ffff8ab3d22fc000 x1874671280830208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828985.702507:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280830208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828985.702517:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828985.702520:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828985.702524:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280830208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828985.702532:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280830336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828985.702552:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828985.702565:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828985/real 1787828985] req@ffff8ab3d22fda40 x1874671280830336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828985.702579:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280830336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828985.702589:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828985.702593:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828985.702597:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280830336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828985.702604:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280830464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828985.702635:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828985.702646:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828985/real 1787828985] req@ffff8ab3d22fcb40 x1874671280830464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828985.702661:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280830464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828985.702672:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828985.702676:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828985.702680:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280830464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828985.702687:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280830592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828985.702705:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828985.702732:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828985/real 1787828985] req@ffff8ab3d22fd680 x1874671280830592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828985.702756:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280830592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828985.702768:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828985.702772:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828985.702776:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280830592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787828986.726057:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828987.750086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828988.774212:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828989.798016:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828990.822009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828990.822019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828990.822024:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828990.822034:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3540 00000100:00080000:1.0:1787828990.822045:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828990.822053:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787828990.822097:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280830720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828990.822125:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828990.822128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828990.822129:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828990.822132:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3540 00000100:00080000:1.0:1787828990.822134:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828990.822137:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828990.822147:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828990.822157:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280830848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828990.822164:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828990.822167:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828990.822168:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828990.822171:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3540 00000100:00080000:1.0:1787828990.822176:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828990.822179:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828990.822184:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828990.822196:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280830976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828990.822197:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280830720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787828990.822224:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828990.822227:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828990.822228:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828990.822230:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3540 00000100:00080000:1.0:1787828990.822232:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828990.822235:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828990.822238:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787828990.822247:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280831104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828990.822253:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828990.822255:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828990.822256:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828990.822260:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3540 00000100:00080000:1.0:1787828990.822277:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828990.822281:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787828990.822285:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:2.0:1787828990.822286:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787828990.822304:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280831232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828990.822342:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828990/real 1787828990] req@ffff8ab3d22fed00 x1874671280830720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828990.822360:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280830720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828990.822369:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828990.822373:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828990.822378:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280830720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828990.822385:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280830848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828990.822401:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828990.822410:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828990/real 1787828990] req@ffff8ab3d22fc780 x1874671280830848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828990.822418:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280830848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828990.822424:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828990.822428:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828990.822430:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280830848:192.168.203.124@tcp:38:kworker.0 00010000:00010000:0.0:1787828990.822446:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828990.822467:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280830976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828990.822481:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828990.822488:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828990/real 1787828990] req@ffff8ab3d22fe1c0 x1874671280830976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828990.822496:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280830976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828990.822502:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828990.822504:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828990.822507:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280830976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828990.822511:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280831104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828990.822521:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828990.822526:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828990/real 1787828990] req@ffff8ab3d22fde00 x1874671280831104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828990.822533:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280831104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828990.822541:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828990.822542:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828990.822544:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280831104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828990.822548:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280831232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828990.822558:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828990.822564:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828990/real 1787828990] req@ffff8ab3d22fd2c0 x1874671280831232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828990.822571:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280831232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828990.822578:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828990.822579:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828990.822581:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280831232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787828991.846065:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828992.870104:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828993.894040:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828994.918129:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828995.942030:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828995.942046:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1787828995.942050:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828995.942052:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828995.942069:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3545 00000100:00080000:0.0:1787828995.942082:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828995.942093:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787828995.942145:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280831360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828995.942183:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828995.942187:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828995.942190:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828995.942194:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3545 00000100:00080000:0.0:1787828995.942198:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828995.942203:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828995.942216:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828995.942232:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280831488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828995.942244:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828995.942250:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828995.942250:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280831360:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787828995.942254:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828995.942264:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3545 00000100:00080000:0.0:1787828995.942268:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828995.942282:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828995.942288:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828995.942303:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280831616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828995.942314:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828995.942318:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828995.942321:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828995.942325:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3545 00000100:00080000:0.0:1787828995.942330:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828995.942335:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787828995.942340:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828995.942347:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787828995.942361:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280831744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828995.942371:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828995.942375:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828995.942377:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828995.942382:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3545 00000100:00080000:0.0:1787828995.942395:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828995.942400:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828995/real 1787828995] req@ffff8ab3f075f840 x1874671280831360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828995.942404:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787828995.942414:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787828995.942432:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280831872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828995.942439:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280831360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828995.942454:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828995.942459:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828995.942465:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280831360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787828995.942494:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280831488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828995.942516:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828995.942530:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828995/real 1787828995] req@ffff8ab3f075e940 x1874671280831488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828995.942543:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280831488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828995.942560:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828995.942563:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828995.942567:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280831488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828995.942573:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280831616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787828995.942590:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828995.942600:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828995/real 1787828995] req@ffff8ab3f075ed00 x1874671280831616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828995.942611:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280831616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828995.942620:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828995.942624:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828995.942627:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280831616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787828995.942633:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280831744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828995.942667:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828995.942678:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828995/real 1787828995] req@ffff8ab3f075da40 x1874671280831744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828995.942691:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280831744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828995.942702:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828995.942705:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828995.942709:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280831744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787828995.942716:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280831872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787828995.942733:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787828995.942742:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828995/real 1787828995] req@ffff8ab3f075f480 x1874671280831872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828995.942754:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280831872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828995.942764:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828995.942767:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787828995.942771:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280831872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787828996.966065:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787828997.990048:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787828999.014036:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829000.038151:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829001.062030:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829001.062053:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829001.062058:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829001.062068:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3550 00000100:00080000:0.0:1787829001.062080:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829001.062103:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829001.062115:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829001.062164:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280832000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829001.062186:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829001.062189:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829001.062190:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829001.062194:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3550 00000100:00080000:0.0:1787829001.062198:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829001.062202:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829001.062214:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280832000:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787829001.062223:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829001.062243:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280832128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829001.062255:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829001.062259:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829001.062261:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829001.062266:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3550 00000100:00080000:0.0:1787829001.062284:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829001.062288:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829001.062292:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829001.062301:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280832256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829001.062311:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829001.062311:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829001.062317:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829001.062318:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829001.062322:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3550 00000100:00080000:0.0:1787829001.062325:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829001.062329:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829001.062332:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829001.062339:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280832384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829001.062346:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829001.062356:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829001.062358:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829001.062364:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3550 00000100:00100000:2.0:1787829001.062366:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829001/real 1787829001] req@ffff8ab3f075e580 x1874671280832000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829001.062368:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829001.062373:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829001.062379:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829001.062393:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280832512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829001.062398:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280832000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829001.062413:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829001.062417:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829001.062420:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280832000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829001.062460:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280832128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829001.062493:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829001.062509:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829001/real 1787829001] req@ffff8ab3f075f480 x1874671280832128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829001.062524:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280832128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829001.062534:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829001.062537:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829001.062541:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280832128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829001.062547:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280832256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829001.062565:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829001.062575:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829001/real 1787829001] req@ffff8ab3f075da40 x1874671280832256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829001.062588:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280832256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829001.062597:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829001.062600:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829001.062604:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280832256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829001.062622:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280832384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829001.062640:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829001.062648:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829001/real 1787829001] req@ffff8ab3f075ed00 x1874671280832384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829001.062667:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280832384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829001.062677:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829001.062681:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829001.062684:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280832384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829001.062691:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280832512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829001.062711:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829001.062722:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829001/real 1787829001] req@ffff8ab3f075e940 x1874671280832512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829001.062735:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280832512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829001.062745:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829001.062749:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829001.062753:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280832512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829002.086064:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829003.110051:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829004.134074:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829005.158079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829006.182008:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829006.182020:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829006.182025:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829006.182039:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3555 00000100:00080000:0.0:1787829006.182053:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829006.182063:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829006.182066:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829006.182114:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280832640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829006.182136:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829006.182139:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829006.182141:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829006.182144:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3555 00000100:00080000:0.0:1787829006.182147:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829006.182150:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829006.182160:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829006.182169:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280832768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829006.182180:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280832640:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829006.182184:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829006.182189:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829006.182190:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829006.182195:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3555 00000100:00080000:0.0:1787829006.182197:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829006.182199:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829006.182203:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829006.182212:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280832896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829006.182218:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829006.182221:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829006.182223:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829006.182225:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3555 00000100:00080000:0.0:1787829006.182243:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829006.182246:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829006.182250:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787829006.182255:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829006.182259:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280833024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829006.182267:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829006.182270:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829006.182272:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829006.182276:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3555 00000100:00080000:0.0:1787829006.182281:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829006.182286:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829006.182292:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829006.182305:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280833152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829006.182318:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829006/real 1787829006] req@ffff8ab3f075f840 x1874671280832640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829006.182336:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280832640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829006.182346:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829006.182349:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829006.182352:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280832640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829006.182381:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280832768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829006.182398:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829006.182408:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829006/real 1787829006] req@ffff8ab3f075cb40 x1874671280832768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829006.182416:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280832768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829006.182421:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829006.182423:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829006.182426:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280832768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829006.182435:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280832896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829006.182447:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829006.182456:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829006/real 1787829006] req@ffff8ab3f075f0c0 x1874671280832896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829006.182469:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280832896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829006.182477:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829006.182481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829006.182484:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280832896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829006.182491:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280833024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829006.182511:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829006.182524:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829006/real 1787829006] req@ffff8ab3f075de00 x1874671280833024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829006.182536:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280833024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829006.182543:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829006.182545:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829006.182548:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280833024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829006.182552:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280833152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829006.182566:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829006.182586:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829006/real 1787829006] req@ffff8ab3f075cf00 x1874671280833152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829006.182596:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280833152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829006.182603:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829006.182605:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829006.182607:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280833152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829007.206033:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829008.230029:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829009.254065:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829010.278030:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829011.301989:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829011.302000:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829011.302004:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829011.302014:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3560 00000100:00080000:2.0:1787829011.302025:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829011.302033:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829011.302035:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829011.302076:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280833280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829011.302101:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829011.302104:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829011.302105:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829011.302108:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3560 00000100:00080000:2.0:1787829011.302110:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829011.302112:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829011.302123:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829011.302133:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280833408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829011.302140:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829011.302143:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829011.302144:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829011.302147:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3560 00000100:00080000:2.0:1787829011.302149:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829011.302151:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829011.302155:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829011.302161:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280833280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829011.302166:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280833536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829011.302178:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829011.302180:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829011.302181:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829011.302183:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3560 00000100:00080000:2.0:1787829011.302185:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829011.302188:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829011.302192:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829011.302199:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280833664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829011.302206:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829011.302209:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829011.302210:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829011.302213:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3560 00000100:00080000:2.0:1787829011.302238:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829011.302243:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829011.302248:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787829011.302252:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829011.302268:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280833792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829011.302323:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829011/real 1787829011] req@ffff8ab3f043b840 x1874671280833280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829011.302346:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280833280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829011.302359:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829011.302364:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829011.302369:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280833280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829011.302378:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280833408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829011.302403:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829011.302419:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829011/real 1787829011] req@ffff8ab3f043b480 x1874671280833408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829011.302432:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280833408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829011.302441:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829011.302445:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829011.302449:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280833408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829011.302479:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280833536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829011.302499:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829011.302509:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829011/real 1787829011] req@ffff8ab3f0439e00 x1874671280833536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829011.302517:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280833536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829011.302523:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829011.302525:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829011.302528:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280833536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829011.302531:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280833664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829011.302557:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829011.302564:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829011/real 1787829011] req@ffff8ab3f043a1c0 x1874671280833664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829011.302572:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280833664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829011.302579:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829011.302581:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829011.302583:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280833664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829011.302587:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280833792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829011.302598:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829011.302604:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829011/real 1787829011] req@ffff8ab3f043a940 x1874671280833792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829011.302622:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280833792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829011.302629:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829011.302631:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829011.302633:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280833792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829012.326039:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829013.350034:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829014.374080:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829015.398060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829016.422021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829016.422036:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829016.422042:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829016.422057:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3566 00000100:00080000:2.0:1787829016.422071:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829016.422081:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829016.422152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280833920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829016.422193:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1787829016.422193:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829016.422202:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829016.422206:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829016.422214:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3566 00000100:00080000:2.0:1787829016.422220:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829016.422226:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829016.422243:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829016.422249:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280833920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829016.422271:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280834048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829016.422285:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829016.422298:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829016.422300:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829016.422305:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3566 00000100:00080000:2.0:1787829016.422340:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829016.422345:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829016.422351:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829016.422365:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280834176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829016.422399:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829016.422405:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829016.422409:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829016.422415:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3566 00000100:00080000:2.0:1787829016.422421:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829016.422426:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280834048:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787829016.422433:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829016.422445:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829016.422470:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280834176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829016.422472:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280834304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829016.422495:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829016.422520:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829016.422524:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829016.422530:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3566 00000100:00080000:2.0:1787829016.422535:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829016.422541:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829016.422549:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280834304:192.168.203.124@tcp:8:kworker.0 00000008:00100000:2.0:1787829016.422557:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829016.422579:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280834432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829016.422616:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280834432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829017.446133:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829018.470090:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829019.494168:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787829019.494347:0:10944:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787829019.494424:0:10944:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829019.494549:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829019.494624:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829019.494640:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829019.494652:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829019.494664:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787829019.494831:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829016/real 1787829019] req@ffff8ab3f043bc00 x1874671280834432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829019.494866:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280834432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829019.494922:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829019.494930:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829019.494937:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280834432:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829019.494949:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829016/real 1787829019] req@ffff8ab3f04383c0 x1874671280834304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829019.494970:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280834304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829019.494994:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829019.494999:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829019.495004:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280834304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829019.495013:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829016/real 1787829019] req@ffff8ab3f043ad00 x1874671280834048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829019.495065:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280834048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829019.495080:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829019.495086:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829019.495328:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280834048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829019.495345:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829016/real 1787829019] req@ffff8ab3f04392c0 x1874671280834176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829019.495367:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280834176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829019.495391:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829019.495397:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829019.495402:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280834176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829019.495413:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829016/real 1787829019] req@ffff8ab3f0439a40 x1874671280833920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829019.495433:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280833920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829019.495446:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829019.495451:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829019.495456:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280833920:192.168.203.124@tcp:250:kworker.0 00010000:00010000:3.0:1787829020.518109:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829021.542017:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829021.542028:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829021.542033:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829021.542044:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3571 00000100:00080000:2.0:1787829021.542055:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829021.542063:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829021.542087:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829021.542105:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280834560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829021.542132:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829021.542137:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829021.542139:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829021.542144:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3571 00000100:00080000:2.0:1787829021.542148:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829021.542153:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829021.542164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829021.542177:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280834560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829021.542183:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280834688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829021.542197:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829021.542201:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829021.542204:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829021.542208:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3571 00000100:00080000:2.0:1787829021.542213:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829021.542219:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829021.542225:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829021.542237:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280834816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829021.542247:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829021.542249:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829021.542254:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829021.542257:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829021.542263:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3571 00000100:00080000:2.0:1787829021.542267:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829021.542272:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829021.542277:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829021.542289:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280834944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829021.542306:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829021.542308:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829021/real 1787829021] req@ffff8ab3f0439680 x1874671280834560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829021.542311:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829021.542313:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829021.542319:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3571 00000100:00080000:2.0:1787829021.542322:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829021.542331:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829021.542341:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829021.542354:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280834560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829021.542357:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280835072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829021.542382:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829021.542393:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829021.542401:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280834560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829021.542429:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280834688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829021.542457:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829021.542487:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829021/real 1787829021] req@ffff8ab3f043a580 x1874671280834688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829021.542501:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280834688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829021.542509:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829021.542513:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829021.542516:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280834688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829021.542522:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280834816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829021.542558:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829021.542570:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829021/real 1787829021] req@ffff8ab3f0438f00 x1874671280834816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829021.542581:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280834816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829021.542588:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829021.542592:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829021.542595:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280834816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829021.542600:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280834944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829021.542621:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829021.542632:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829021/real 1787829021] req@ffff8ab3f0438b40 x1874671280834944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829021.542643:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280834944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829021.542652:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829021.542655:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829021.542659:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280834944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829021.542665:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280835072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829021.542684:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829021.542695:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829021/real 1787829021] req@ffff8ab3f0438780 x1874671280835072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829021.542706:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280835072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829021.542715:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829021.542718:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829021.542721:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280835072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829022.566065:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829023.590086:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829024.614021:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829025.638065:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829026.662039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829026.662054:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829026.662059:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829026.662075:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3576 00000100:00080000:0.0:1787829026.662088:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829026.662099:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829026.662119:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829026.662150:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280835200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829026.662193:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829026.662196:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829026.662198:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829026.662201:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3576 00000100:00080000:0.0:1787829026.662203:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829026.662206:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829026.662217:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829026.662228:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280835328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829026.662236:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829026.662239:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829026.662241:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829026.662245:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3576 00000100:00080000:0.0:1787829026.662248:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829026.662252:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829026.662255:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829026.662262:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280835456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829026.662271:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829026.662274:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829026.662275:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829026.662279:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3576 00000100:00080000:0.0:1787829026.662283:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829026.662290:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280835200:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829026.662291:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829026.662303:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829026.662314:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280835584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829026.662321:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829026.662324:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829026.662326:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829026.662331:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3576 00000100:00080000:0.0:1787829026.662334:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829026.662337:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829026.662342:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829026.662352:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280835712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829026.662376:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829026.662421:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829026/real 1787829026] req@ffff8ab3f075c000 x1874671280835200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829026.662563:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280835200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829026.662575:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829026.662579:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829026.662582:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280835200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829026.662590:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280835328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829026.663867:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829026.663906:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829026/real 1787829026] req@ffff8ab3f075cf00 x1874671280835328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829026.663920:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280835328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829026.663934:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829026.663937:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829026.663940:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280835328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829026.663964:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280835456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829026.663989:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829026.664003:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829026/real 1787829026] req@ffff8ab3f075de00 x1874671280835456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829026.664019:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280835456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829026.664030:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829026.664035:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829026.664039:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280835456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829026.664048:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280835584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829026.664072:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829026.664088:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829026/real 1787829026] req@ffff8ab3f075f0c0 x1874671280835584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829026.664104:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280835584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829026.664119:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829026.664124:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829026.664130:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280835584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829026.664136:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280835712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829026.664156:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829026.664165:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829026/real 1787829026] req@ffff8ab3f075cb40 x1874671280835712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829026.664175:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280835712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829026.664183:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829026.664185:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829026.664188:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280835712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829027.686060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829028.710061:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829029.734104:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829030.758077:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829031.782022:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829031.782037:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829031.782043:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829031.782057:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3581 00000100:00080000:0.0:1787829031.782072:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829031.782081:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829031.782109:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829031.782127:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280835840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829031.782153:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829031.782156:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829031.782157:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829031.782160:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3581 00000100:00080000:0.0:1787829031.782164:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829031.782179:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829031.782189:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829031.782200:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280835968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829031.782214:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829031.782214:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280835840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829031.782231:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829031.782233:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829031.782240:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3581 00000100:00080000:0.0:1787829031.782245:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829031.782252:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829031.782258:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829031.782277:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280836096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829031.782289:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829031.782294:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829031.782303:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829031.782309:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829031.782316:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3581 00000100:00080000:0.0:1787829031.782321:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829031.782338:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829031.782349:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829031.782362:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280836224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829031.782371:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829031.782372:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829031/real 1787829031] req@ffff8ab3f075f840 x1874671280835840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829031.782378:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829031.782384:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829031.782412:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3581 00000100:00080000:0.0:1787829031.782418:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829031.782418:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280835840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829031.782431:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829031.782448:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787829031.782451:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829031.782457:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829031.782462:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280835840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829031.782465:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280836352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829031.782499:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280835968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829031.782524:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829031.782534:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829031/real 1787829031] req@ffff8ab3f075e940 x1874671280835968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829031.782542:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280835968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829031.782548:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829031.782551:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829031.782553:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280835968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829031.782557:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280836096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829031.782568:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829031.782574:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829031/real 1787829031] req@ffff8ab3f075ed00 x1874671280836096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829031.782581:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280836096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829031.782586:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829031.782588:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829031.782590:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280836096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829031.782594:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280836224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829031.782604:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829031.782610:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829031/real 1787829031] req@ffff8ab3f075da40 x1874671280836224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829031.782616:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280836224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829031.782624:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829031.782626:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829031.782628:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280836224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829031.782632:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280836352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829031.782641:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829031.782645:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829031/real 1787829031] req@ffff8ab3f075f480 x1874671280836352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829031.782652:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280836352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829031.782658:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829031.782660:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829031.782662:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280836352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829032.806072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829033.830069:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829034.854090:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829035.878053:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829036.901991:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829036.902000:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829036.902004:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829036.902013:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3586 00000100:00080000:2.0:1787829036.902023:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829036.902030:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829036.902042:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829036.902064:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280836480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829036.902086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829036.902089:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829036.902091:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829036.902094:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3586 00000100:00080000:2.0:1787829036.902097:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829036.902099:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829036.902108:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829036.902119:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280836608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829036.902125:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280836480:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829036.902130:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829036.902134:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829036.902135:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829036.902138:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3586 00000100:00080000:2.0:1787829036.902140:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829036.902142:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829036.902147:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829036.902155:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280836736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829036.902163:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829036.902165:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829036.902167:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829036.902169:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3586 00000100:00080000:2.0:1787829036.902179:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829036.902182:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829036.902185:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829036.902192:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829036.902194:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280836864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829036.902202:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829036.902205:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829036.902206:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829036.902209:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3586 00000100:00080000:2.0:1787829036.902211:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829036.902214:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829036.902219:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829036.902230:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280836992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829036.902248:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829036/real 1787829036] req@ffff8ab3f043b0c0 x1874671280836480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829036.902268:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280836480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829036.902276:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829036.902279:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829036.902282:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280836480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829036.902304:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280836608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829036.902328:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829036.902343:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829036/real 1787829036] req@ffff8ab3f0438780 x1874671280836608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829036.902357:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280836608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829036.902382:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829036.902386:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829036.902390:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280836608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829036.902396:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280836736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829036.902417:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829036.902430:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829036/real 1787829036] req@ffff8ab3f0438b40 x1874671280836736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829036.902439:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280836736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829036.902444:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829036.902447:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829036.902449:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280836736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829036.902453:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280836864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829036.902466:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829036.902473:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829036/real 1787829036] req@ffff8ab3f0438f00 x1874671280836864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829036.902480:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280836864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829036.902495:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829036.902498:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829036.902500:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280836864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829036.902506:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280836992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829036.902517:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829036.902524:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829036/real 1787829036] req@ffff8ab3f043a580 x1874671280836992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829036.902531:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280836992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829036.902537:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829036.902539:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829036.902542:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280836992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829037.926098:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829038.950142:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829039.974093:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829040.998088:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829042.021963:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829042.021971:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829042.021976:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829042.021984:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3591 00000100:00080000:2.0:1787829042.021998:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829042.022004:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829042.022026:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829042.022038:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280837120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829042.022061:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829042.022063:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829042.022064:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829042.022067:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3591 00000100:00080000:2.0:1787829042.022070:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829042.022073:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829042.022081:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829042.022093:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280837248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829042.022100:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280837120:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829042.022103:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829042.022110:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829042.022112:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829042.022115:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3591 00000100:00080000:2.0:1787829042.022117:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829042.022120:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829042.022123:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829042.022131:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280837376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829042.022144:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829042.022146:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829042.022148:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829042.022151:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3591 00000100:00080000:2.0:1787829042.022153:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829042.022156:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829042.022159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829042.022164:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829042.022171:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280837504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829042.022179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829042.022182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829042.022183:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829042.022186:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3591 00000100:00080000:2.0:1787829042.022188:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829042.022190:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829042.022193:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829042.022212:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280837632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829042.022220:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829042/real 1787829042] req@ffff8ab3f0439680 x1874671280837120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829042.022236:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280837120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829042.022245:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829042.022248:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829042.022252:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280837120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829042.022281:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280837248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829042.022295:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829042.022304:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829042/real 1787829042] req@ffff8ab3f0439a40 x1874671280837248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829042.022320:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280837248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829042.022342:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829042.022345:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829042.022347:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280837248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829042.022351:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280837376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829042.022366:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829042.022374:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829042/real 1787829042] req@ffff8ab3f04392c0 x1874671280837376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829042.022381:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280837376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829042.022386:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829042.022389:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829042.022391:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280837376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829042.022395:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280837504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829042.022406:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829042.022411:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829042/real 1787829042] req@ffff8ab3f043ad00 x1874671280837504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829042.022418:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280837504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829042.022424:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829042.022426:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829042.022427:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280837504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829042.022431:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280837632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829042.022441:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829042.022446:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829042/real 1787829042] req@ffff8ab3f04383c0 x1874671280837632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829042.022456:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280837632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829042.022466:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829042.022470:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829042.022473:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280837632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829043.046061:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829044.070064:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829045.094089:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829046.118057:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829047.142007:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829047.142018:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829047.142023:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829047.142035:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3596 00000100:00080000:2.0:1787829047.142046:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829047.142054:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829047.142071:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829047.142094:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280837760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829047.142119:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829047.142123:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829047.142125:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829047.142131:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3596 00000100:00080000:2.0:1787829047.142136:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829047.142141:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829047.142150:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280837760:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787829047.142152:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829047.142172:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280837888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829047.142184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829047.142188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829047.142191:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829047.142196:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3596 00000100:00080000:2.0:1787829047.142201:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829047.142205:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829047.142210:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829047.142210:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829047.142226:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280838016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829047.142234:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829047.142237:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829047.142239:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829047.142243:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3596 00000100:00080000:2.0:1787829047.142246:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829047.142250:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829047.142253:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829047.142257:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829047/real 1787829047] req@ffff8ab3f043bc00 x1874671280837760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829047.142265:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280838144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829047.142274:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829047.142278:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280837760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829047.142279:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829047.142282:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829047.142303:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3596 00000100:00080000:1.0:1787829047.142307:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829047.142308:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829047.142316:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787829047.142317:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829047.142323:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280837760:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787829047.142324:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829047.142339:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280838272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829047.142351:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280837888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829047.142371:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829047.142381:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829047/real 1787829047] req@ffff8ab3f043a940 x1874671280837888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829047.142390:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280837888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829047.142396:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829047.142398:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829047.142400:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280837888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829047.142404:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280838016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829047.142414:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829047.142426:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829047/real 1787829047] req@ffff8ab3f043a1c0 x1874671280838016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829047.142433:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280838016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829047.142443:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829047.142446:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829047.142448:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280838016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829047.142451:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280838144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829047.142462:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829047.142469:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829047/real 1787829047] req@ffff8ab3f0439e00 x1874671280838144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829047.142478:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280838144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829047.142488:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829047.142492:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829047.142495:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280838144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829047.142501:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280838272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829047.142520:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829047.142533:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829047/real 1787829047] req@ffff8ab3f043b480 x1874671280838272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829047.142546:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280838272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829047.142556:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829047.142558:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829047.142561:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280838272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829048.166047:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829049.190098:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829050.214103:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829051.238077:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829052.262001:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829052.262012:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829052.262016:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829052.262027:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3601 00000100:00080000:0.0:1787829052.262038:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829052.262045:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829052.262091:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280838400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787829052.262093:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829052.262116:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829052.262119:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829052.262121:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829052.262124:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3601 00000100:00080000:0.0:1787829052.262126:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829052.262129:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829052.262139:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829052.262151:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280838528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829052.262161:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829052.262162:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280838400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829052.262165:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829052.262169:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829052.262175:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3601 00000100:00080000:0.0:1787829052.262178:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829052.262181:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829052.262184:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829052.262193:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280838656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829052.262199:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829052.262202:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829052.262204:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829052.262207:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3601 00000100:00080000:0.0:1787829052.262210:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829052.262213:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829052.262216:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829052.262224:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280838784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829052.262230:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829052.262233:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829052.262234:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829052.262237:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3601 00000100:00080000:0.0:1787829052.262242:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829052.262245:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829052.262248:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829052.262274:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280838912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829052.262275:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829052.262341:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829052/real 1787829052] req@ffff8ab3f075e580 x1874671280838400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829052.262364:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280838400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829052.262378:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829052.262383:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829052.262388:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280838400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829052.262426:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280838528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829052.262453:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829052.262468:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829052/real 1787829052] req@ffff8ab3f075f480 x1874671280838528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829052.262482:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280838528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829052.262492:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829052.262496:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829052.262500:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280838528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829052.262508:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280838656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829052.262527:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829052.262554:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829052/real 1787829052] req@ffff8ab3f075da40 x1874671280838656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829052.262568:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280838656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829052.262578:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829052.262582:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829052.262586:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280838656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829052.262593:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280838784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829052.262614:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829052.262626:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829052/real 1787829052] req@ffff8ab3f075ed00 x1874671280838784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829052.262640:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280838784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829052.262651:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829052.262664:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829052.262669:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280838784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829052.262676:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280838912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829052.262698:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829052.262711:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829052/real 1787829052] req@ffff8ab3f075e940 x1874671280838912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829052.262724:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280838912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829052.262736:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829052.262740:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829052.262744:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280838912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829053.286124:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829054.310073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829055.334104:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829056.358029:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829057.381978:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829057.381989:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829057.381993:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829057.382002:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3607 00000100:00080000:0.0:1787829057.382011:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829057.382017:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829057.382059:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280839040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829057.382080:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829057.382083:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829057.382085:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1787829057.382086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829057.382091:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3607 00000100:00080000:0.0:1787829057.382095:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829057.382098:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829057.382106:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829057.382121:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280839168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829057.382126:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280839040:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829057.382131:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829057.382135:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829057.382137:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829057.382140:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3607 00000100:00080000:0.0:1787829057.382142:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829057.382144:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829057.382148:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829057.382157:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280839296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829057.382164:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829057.382166:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829057.382167:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829057.382170:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3607 00000100:00080000:0.0:1787829057.382187:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829057.382191:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829057.382195:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829057.382199:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829057.382204:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280839424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829057.382212:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829057.382215:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829057.382231:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829057.382235:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3607 00000100:00080000:0.0:1787829057.382238:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829057.382241:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829057.382246:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829057.382255:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280839552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829057.382278:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829057/real 1787829057] req@ffff8ab3f075f840 x1874671280839040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829057.382298:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280839040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829057.382309:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829057.382313:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829057.382317:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280839040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829057.382340:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280839168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829057.382364:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829057.382379:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829057/real 1787829057] req@ffff8ab3f075cb40 x1874671280839168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829057.382391:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280839168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829057.382400:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829057.382404:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829057.382407:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280839168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829057.382414:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280839296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829057.382436:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829057.382448:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829057/real 1787829057] req@ffff8ab3f075f0c0 x1874671280839296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829057.382461:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280839296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829057.382469:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829057.382472:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829057.382475:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280839296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829057.382481:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280839424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829057.382503:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829057.382526:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829057/real 1787829057] req@ffff8ab3f075de00 x1874671280839424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829057.382538:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280839424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829057.382549:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829057.382552:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829057.382555:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280839424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829057.382562:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280839552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829057.382582:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829057.382594:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829057/real 1787829057] req@ffff8ab3f075cf00 x1874671280839552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829057.382605:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280839552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829057.382614:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829057.382617:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829057.382620:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280839552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829058.406165:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829059.430078:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829060.454094:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829061.478065:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829062.501993:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829062.502000:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829062.502006:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829062.502016:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3612 00000100:00080000:0.0:1787829062.502026:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829062.502034:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829062.502060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829062.502077:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280839680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829062.502101:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829062.502104:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829062.502106:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829062.502109:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3612 00000100:00080000:0.0:1787829062.502112:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829062.502116:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829062.502125:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829062.502137:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280839808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829062.502148:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280839680:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829062.502151:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829062.502164:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829062.502166:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829062.502169:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3612 00000100:00080000:0.0:1787829062.502172:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829062.502174:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829062.502177:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829062.502202:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280839936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829062.502215:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829062.502219:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829062.502221:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829062.502225:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3612 00000100:00080000:0.0:1787829062.502240:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829062.502243:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829062.502247:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829062.502256:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829062.502257:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280840064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829062.502270:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829062.502273:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829062.502275:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829062.502279:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3612 00000100:00080000:0.0:1787829062.502283:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829062.502286:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829062.502289:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829062.502299:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280840192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829062.502330:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829062/real 1787829062] req@ffff8ab3f075c000 x1874671280839680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829062.502348:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280839680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829062.502358:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829062.502362:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829062.502365:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280839680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829062.502389:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280839808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829062.502407:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829062.502416:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829062/real 1787829062] req@ffff8ab3f075cf00 x1874671280839808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829062.502424:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280839808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829062.502434:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829062.502437:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829062.502439:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280839808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829062.502443:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280839936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829062.502453:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829062.502459:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829062/real 1787829062] req@ffff8ab3f075de00 x1874671280839936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829062.502466:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280839936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829062.502472:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829062.502473:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829062.502475:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280839936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829062.502479:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280840064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829062.502490:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829062.502495:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829062/real 1787829062] req@ffff8ab3f075f0c0 x1874671280840064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829062.502503:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280840064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829062.502510:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829062.502511:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829062.502514:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280840064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829062.502518:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280840192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829062.502528:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829062.502534:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829062/real 1787829062] req@ffff8ab3f075cb40 x1874671280840192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829062.502541:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280840192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829062.502547:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829062.502549:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829062.502551:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280840192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829063.526087:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829064.550125:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829065.574063:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829066.598053:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829067.622197:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829067.622212:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829067.622217:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1787829067.622221:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829067.622250:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3617 00000100:00080000:3.0:1787829067.622265:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829067.622275:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829067.622349:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280840320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829067.622399:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829067.622403:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829067.622406:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829067.622410:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3617 00000100:00080000:3.0:1787829067.622417:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829067.622423:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829067.622437:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829067.622450:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280840320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829067.622455:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280840448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829067.622477:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829067.622482:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829067.622484:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829067.622490:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3617 00000100:00080000:3.0:1787829067.622493:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829067.622499:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829067.622505:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829067.622520:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280840576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829067.622531:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829067.622543:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829067.622562:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829067.622566:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829067.622574:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3617 00000100:00080000:3.0:1787829067.622581:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829067.622587:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829067.622595:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829067.622610:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280840704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829067.622625:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829067/real 1787829067] req@ffff8ab3d0a41e00 x1874671280840320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829067.622667:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829067.622678:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829067.622681:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829067.622689:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3617 00000100:00080000:3.0:1787829067.622692:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829067.622694:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280840320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829067.622720:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787829067.622724:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829067.622730:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787829067.622735:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829067.622748:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280840320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829067.622757:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280840832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829067.622790:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280840448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829067.622815:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829067.622836:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829067/real 1787829067] req@ffff8ab3d0a430c0 x1874671280840448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829067.622847:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280840448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829067.622854:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829067.622856:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829067.622859:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280840448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829067.622864:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280840576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829067.622879:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829067.622940:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829067/real 1787829067] req@ffff8ab3d0a43c00 x1874671280840576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829067.622949:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280840576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829067.622956:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829067.622959:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829067.622961:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280840576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829067.622992:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280840704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829067.623009:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829067.623016:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829067/real 1787829067] req@ffff8ab3d0a43480 x1874671280840704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829067.623024:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280840704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829067.623032:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829067.623035:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829067.623037:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280840704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829067.623041:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280840832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829067.623068:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829067.623076:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829067/real 1787829067] req@ffff8ab3d0a412c0 x1874671280840832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829067.623085:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280840832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829067.623093:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829067.623095:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829067.623097:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280840832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829068.646028:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829069.670047:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829070.694151:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829071.718068:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829072.741994:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829072.742003:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829072.742007:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829072.742015:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3622 00000100:00080000:1.0:1787829072.742024:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829072.742047:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829072.742084:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280840960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829072.742123:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829072.742127:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829072.742129:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829072.742131:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3622 00000100:00080000:1.0:1787829072.742134:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829072.742138:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829072.742147:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829072.742158:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280841088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829072.742162:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280840960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787829072.742175:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829072.742178:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829072.742179:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829072.742183:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3622 00000100:00080000:1.0:1787829072.742186:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829072.742188:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829072.742192:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829072.742202:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280841216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829072.742209:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829072.742211:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829072.742213:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829072.742216:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3622 00000100:00080000:1.0:1787829072.742218:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829072.742221:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829072.742225:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787829072.742226:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829072.742237:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280841344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829072.742246:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829072.742248:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829072.742250:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829072.742252:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3622 00000100:00080000:1.0:1787829072.742254:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829072.742257:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829072.742260:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829072.742270:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280841472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829072.742276:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829072/real 1787829072] req@ffff8ab3d22fd2c0 x1874671280840960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829072.742303:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280840960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829072.742317:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829072.742322:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829072.742328:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280840960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829072.742357:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280841088:192.168.203.124@tcp:38:kworker.0 00010000:00010000:3.0:1787829072.742365:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1787829072.742381:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829072.742397:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829072/real 1787829072] req@ffff8ab3d22fde00 x1874671280841088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829072.742413:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280841088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829072.742424:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829072.742428:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829072.742432:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280841088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829072.742437:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280841216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829072.742454:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829072.742462:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829072/real 1787829072] req@ffff8ab3d22fe1c0 x1874671280841216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829072.742472:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280841216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829072.742483:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829072.742485:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829072.742488:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280841216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829072.742492:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280841344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829072.742520:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829072.742526:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829072/real 1787829072] req@ffff8ab3d22fc780 x1874671280841344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829072.742533:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280841344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829072.742541:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829072.742542:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829072.742544:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280841344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829072.742548:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280841472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829072.742574:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829072.742581:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829072/real 1787829072] req@ffff8ab3d22fed00 x1874671280841472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829072.742590:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280841472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829072.742597:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829072.742598:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829072.742601:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280841472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829073.766060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829074.790057:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829075.814087:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829076.838068:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829077.862057:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829077.862067:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829077.862071:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829077.862098:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3627 00000100:00080000:2.0:1787829077.862109:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829077.862116:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829077.862131:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829077.862151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280841600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829077.862187:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829077.862190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829077.862192:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829077.862195:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3627 00000100:00080000:2.0:1787829077.862198:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829077.862202:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829077.862211:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829077.862226:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280841728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829077.862227:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280841600:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829077.862244:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829077.862247:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829077.862260:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829077.862264:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3627 00000100:00080000:2.0:1787829077.862268:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829077.862271:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829077.862277:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829077.862287:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280841856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829077.862295:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829077.862298:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829077.862300:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829077.862303:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3627 00000100:00080000:2.0:1787829077.862306:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829077.862310:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829077.862315:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829077.862329:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829077.862339:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280841984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829077.862347:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829077.862351:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829077.862354:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829077.862361:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3627 00000100:00080000:2.0:1787829077.862366:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829077.862371:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829077.862377:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829077/real 1787829077] req@ffff8ab3f043b840 x1874671280841600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829077.862380:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829077.862409:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280842112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829077.862422:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280841600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829077.862465:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829077.862479:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829077.862484:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280841600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829077.862506:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280841728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829077.862528:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829077.862540:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829077/real 1787829077] req@ffff8ab3f043b480 x1874671280841728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829077.862551:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280841728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829077.862558:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829077.862561:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829077.862564:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280841728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829077.862568:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280841856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829077.862581:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829077.862588:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829077/real 1787829077] req@ffff8ab3f0439e00 x1874671280841856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829077.862597:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280841856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829077.862603:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829077.862606:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829077.862608:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280841856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829077.862612:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280841984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829077.862624:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829077.862631:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829077/real 1787829077] req@ffff8ab3f043a1c0 x1874671280841984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829077.862645:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280841984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829077.862653:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829077.862655:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829077.862657:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280841984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829077.862662:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280842112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829077.862673:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829077.862679:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829077/real 1787829077] req@ffff8ab3f043a940 x1874671280842112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829077.862697:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280842112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829077.862705:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829077.862707:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829077.862710:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280842112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829078.886045:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829079.910113:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829080.934110:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829081.958073:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829082.981989:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829082.982001:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829082.982006:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829082.982017:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3632 00000100:00080000:2.0:1787829082.982027:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829082.982037:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829082.982048:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829082.982105:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280842240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829082.982136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829082.982140:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829082.982143:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829082.982147:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3632 00000100:00080000:2.0:1787829082.982152:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829082.982157:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829082.982177:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829082.982183:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280842240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829082.982197:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280842368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829082.982212:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829082.982216:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829082.982218:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829082.982224:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3632 00000100:00080000:2.0:1787829082.982230:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829082.982235:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829082.982242:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829082.982258:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280842496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829082.982270:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829082.982276:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829082.982279:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829082.982282:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280842368:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787829082.982288:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3632 00000100:00080000:2.0:1787829082.982294:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829082.982301:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829082.982308:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829082.982313:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280842496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829082.982329:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280842624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829082.982342:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829082.982346:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829082.982349:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829082.982355:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280842624:192.168.203.124@tcp:8:kworker.0 00000100:00080000:2.0:1787829082.982360:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3632 00000100:00080000:2.0:1787829082.982366:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829082.982373:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829082.982391:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829082.982405:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280842752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829082.982444:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280842752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829084.006103:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829085.030084:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829086.054117:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787829086.054311:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787829086.054350:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829086.054364:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829086.054392:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829086.054402:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829086.054411:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829086.054419:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787829086.054435:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829082/real 1787829086] req@ffff8ab3f04392c0 x1874671280842624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829086.054462:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280842624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829086.054476:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829086.054481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829086.054485:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280842624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829086.054493:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829082/real 1787829086] req@ffff8ab3f04383c0 x1874671280842368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829086.054506:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280842368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829086.054515:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829086.054519:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829086.054523:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280842368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829086.054528:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829082/real 1787829086] req@ffff8ab3f043ad00 x1874671280842496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829086.054539:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280842496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829086.054547:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829086.054550:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829086.054553:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280842496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829086.054561:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829082/real 1787829086] req@ffff8ab3f043bc00 x1874671280842240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829086.054573:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280842240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829086.054581:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829086.054584:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829086.054587:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280842240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829086.054612:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829082/real 1787829086] req@ffff8ab3f0439a40 x1874671280842752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829086.054633:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280842752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829086.054643:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829086.054647:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829086.054650:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280842752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829087.078057:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829088.102125:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829088.102135:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829088.102140:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829088.102151:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3637 00000100:00080000:3.0:1787829088.102162:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787829088.102165:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829088.102171:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829088.102212:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280842880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829088.102274:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829088.102278:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829088.102280:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829088.102284:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3637 00000100:00080000:3.0:1787829088.102288:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829088.102294:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829088.102302:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829088.102313:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280843008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829088.102335:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280842880:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829088.102336:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829088.102344:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829088.102346:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829088.102351:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3637 00000100:00080000:3.0:1787829088.102356:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829088.102360:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829088.102366:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829088.102381:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280843136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829088.102404:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829088.102411:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787829088.102412:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829088.102416:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829088.102423:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3637 00000100:00080000:3.0:1787829088.102429:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829088.102434:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829088.102441:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829088.102457:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280843264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829088.102467:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829088.102470:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829088.102472:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829088.102477:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3637 00000100:00080000:3.0:1787829088.102480:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829088.102485:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829088/real 1787829088] req@ffff8ab3d0a43840 x1874671280842880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829088.102489:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829088.102499:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829088.102513:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280842880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829088.102516:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280843392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829088.102545:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829088.102553:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829088.102558:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280842880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829088.102585:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280843008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829088.102610:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829088.102626:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829088/real 1787829088] req@ffff8ab3d0a40000 x1874671280843008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829088.102639:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280843008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829088.102647:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829088.102650:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829088.102654:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280843008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829088.102660:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280843136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829088.102681:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829088.102693:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829088/real 1787829088] req@ffff8ab3d0a421c0 x1874671280843136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829088.102703:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280843136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829088.102711:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829088.102713:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829088.102716:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280843136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829088.102722:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280843264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829088.102742:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829088.102753:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829088/real 1787829088] req@ffff8ab3d0a412c0 x1874671280843264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829088.102766:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280843264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829088.102776:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829088.102779:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829088.102782:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280843264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829088.102788:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280843392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829088.102807:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829088.102818:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829088/real 1787829088] req@ffff8ab3d0a43480 x1874671280843392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829088.102829:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280843392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829088.102838:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829088.102841:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829088.102844:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280843392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829089.126038:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829090.150111:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829091.174059:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829092.198069:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829093.221997:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829093.222005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829093.222009:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829093.222019:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3642 00000100:00080000:2.0:1787829093.222028:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829093.222034:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829093.222051:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829093.222067:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280843520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829093.222088:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829093.222091:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829093.222093:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829093.222095:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3642 00000100:00080000:2.0:1787829093.222099:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829093.222102:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829093.222110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829093.222123:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280843520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829093.222123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280843648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829093.222144:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829093.222147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829093.222148:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829093.222151:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3642 00000100:00080000:2.0:1787829093.222154:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829093.222157:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829093.222160:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829093.222170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280843776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829093.222176:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829093.222179:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829093.222181:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829093.222183:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3642 00000100:00080000:2.0:1787829093.222185:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829093.222188:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829093.222191:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829093.222192:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829093.222202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280843904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829093.222211:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829093.222214:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829093.222215:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829093.222218:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3642 00000100:00080000:2.0:1787829093.222221:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829093.222224:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829093.222228:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829093.222240:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829093/real 1787829093] req@ffff8ab3f0439680 x1874671280843520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829093.222243:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280844032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829093.222287:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280843520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829093.222296:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829093.222300:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829093.222303:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280843520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829093.222321:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280843648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829093.222336:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829093.222345:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829093/real 1787829093] req@ffff8ab3f043a580 x1874671280843648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829093.222353:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280843648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829093.222359:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829093.222361:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829093.222363:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280843648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829093.222367:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280843776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829093.222377:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829093.222383:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829093/real 1787829093] req@ffff8ab3f0438f00 x1874671280843776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829093.222390:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280843776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829093.222400:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829093.222402:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829093.222404:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280843776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829093.222407:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280843904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829093.222417:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829093.222422:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829093/real 1787829093] req@ffff8ab3f0438b40 x1874671280843904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829093.222429:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280843904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829093.222436:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829093.222438:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829093.222440:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280843904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829093.222443:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280844032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829093.222452:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829093.222457:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829093/real 1787829093] req@ffff8ab3f0438780 x1874671280844032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829093.222464:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280844032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829093.222469:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829093.222471:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829093.222474:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280844032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829094.246058:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829095.270059:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829096.294064:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829097.318051:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829098.341998:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829098.342008:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829098.342012:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829098.342022:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3647 00000100:00080000:0.0:1787829098.342034:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829098.342042:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829098.342048:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829098.342080:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280844160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829098.342102:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829098.342105:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829098.342107:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829098.342110:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3647 00000100:00080000:0.0:1787829098.342113:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829098.342116:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829098.342124:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829098.342135:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280844288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829098.342143:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829098.342146:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829098.342148:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829098.342152:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280844160:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829098.342153:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3647 00000100:00080000:0.0:1787829098.342159:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829098.342164:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829098.342168:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829098.342176:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280844416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829098.342182:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829098.342185:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829098.342186:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829098.342189:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3647 00000100:00080000:0.0:1787829098.342192:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829098.342194:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829098.342197:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829098.342206:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280844544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829098.342213:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829098.342215:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829098.342216:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829098.342219:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3647 00000100:00080000:0.0:1787829098.342222:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829098.342228:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829098.342228:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787829098.342235:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829098.342243:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280844672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829098.342283:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829098/real 1787829098] req@ffff8ab3f075f840 x1874671280844160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829098.342302:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280844160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829098.342314:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829098.342318:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829098.342322:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280844160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829098.342347:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280844288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829098.342372:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829098.342386:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829098/real 1787829098] req@ffff8ab3f075e940 x1874671280844288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829098.342400:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280844288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829098.342409:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829098.342413:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829098.342417:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280844288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829098.342423:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280844416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829098.342444:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829098.342456:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829098/real 1787829098] req@ffff8ab3f075ed00 x1874671280844416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829098.342469:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280844416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829098.342478:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829098.342481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829098.342485:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280844416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829098.342491:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280844544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829098.342511:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829098.342523:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829098/real 1787829098] req@ffff8ab3f075da40 x1874671280844544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829098.342536:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280844544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829098.342546:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829098.342550:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829098.342553:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280844544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829098.342560:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280844672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829098.342579:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829098.342590:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829098/real 1787829098] req@ffff8ab3f075f480 x1874671280844672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829098.342602:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280844672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829098.342613:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829098.342617:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829098.342621:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280844672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829099.366078:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829100.390034:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829101.414078:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829102.438034:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829103.462029:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829103.462055:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829103.462061:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829103.462077:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3653 00010000:00010000:0.0:1787829103.462091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829103.462092:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829103.462117:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829103.462170:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280844800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829103.462199:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829103.462203:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829103.462205:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829103.462210:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3653 00000100:00080000:3.0:1787829103.462214:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829103.462219:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829103.462232:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829103.462237:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280844800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829103.462251:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280844928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829103.462264:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829103.462268:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829103.462270:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829103.462275:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3653 00000100:00080000:3.0:1787829103.462279:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829103.462283:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829103.462290:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829103.462301:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280845056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829103.462311:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829103.462315:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829103.462317:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829103.462321:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3653 00000100:00080000:3.0:1787829103.462325:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829103.462330:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829103.462334:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829103.462342:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829103.462358:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280845184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829103.462368:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829103.462372:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829103.462374:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829103.462379:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3653 00000100:00080000:3.0:1787829103.462383:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829103.462389:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829103.462394:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829103/real 1787829103] req@ffff8ab3d0a43c00 x1874671280844800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829103.462397:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829103.462425:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280844800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829103.462425:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280845312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829103.462448:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829103.462453:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829103.462458:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280844800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829103.462486:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280844928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829103.462512:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829103.462527:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829103/real 1787829103] req@ffff8ab3d0a430c0 x1874671280844928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829103.462542:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280844928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829103.462551:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829103.462555:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829103.462559:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280844928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829103.462567:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280845056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829103.462589:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829103.462600:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829103/real 1787829103] req@ffff8ab3d0a41e00 x1874671280845056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829103.462609:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280845056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829103.462617:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829103.462620:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829103.462623:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280845056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829103.462627:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280845184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829103.462644:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829103.462651:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829103/real 1787829103] req@ffff8ab3d0a43480 x1874671280845184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829103.462659:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280845184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829103.462666:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829103.462668:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829103.462670:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280845184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829103.462673:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280845312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829103.462693:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829103.462700:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829103/real 1787829103] req@ffff8ab3d0a412c0 x1874671280845312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829103.462708:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280845312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829103.462715:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829103.462717:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829103.462719:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280845312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829104.486031:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829105.510101:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829106.534055:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829107.558049:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829108.581983:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829108.581997:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829108.582003:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829108.582018:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3658 00010000:00010000:2.0:1787829108.582029:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829108.582031:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829108.582043:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829108.582094:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280845440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829108.582123:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829108.582126:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829108.582128:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829108.582131:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3658 00000100:00080000:3.0:1787829108.582135:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829108.582139:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829108.582150:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829108.582162:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280845440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829108.582164:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280845568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829108.582176:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829108.582179:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829108.582180:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829108.582183:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3658 00000100:00080000:3.0:1787829108.582186:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829108.582188:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829108.582192:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829108.582202:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280845696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829108.582220:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829108.582224:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829108.582227:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829108.582232:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3658 00000100:00080000:3.0:1787829108.582237:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829108.582239:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829108.582245:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829108.582253:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829108.582267:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280845824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829108.582274:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829108.582278:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829108.582279:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829108.582283:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3658 00000100:00080000:3.0:1787829108.582287:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829108.582290:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829108.582294:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829108/real 1787829108] req@ffff8ab3d0a421c0 x1874671280845440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829108.582296:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829108.582310:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280845952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829108.582316:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280845440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829108.582330:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829108.582334:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829108.582338:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280845440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829108.582359:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280845568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829108.582374:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829108.582383:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829108/real 1787829108] req@ffff8ab3d0a40000 x1874671280845568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829108.582391:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280845568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829108.582402:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829108.582404:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829108.582406:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280845568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829108.582410:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280845696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829108.582420:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829108.582425:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829108/real 1787829108] req@ffff8ab3d0a43840 x1874671280845696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829108.582432:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280845696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829108.582438:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829108.582440:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829108.582442:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280845696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829108.582445:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280845824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829108.582455:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829108.582461:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829108/real 1787829108] req@ffff8ab3d0a412c0 x1874671280845824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829108.582468:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280845824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829108.582475:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829108.582476:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829108.582479:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280845824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829108.582483:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280845952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829108.582492:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829108.582497:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829108/real 1787829108] req@ffff8ab3d0a43480 x1874671280845952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829108.582504:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280845952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829108.582511:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829108.582513:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829108.582514:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280845952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829109.606069:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829110.630086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829111.654056:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829112.678068:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829113.702000:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829113.702011:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829113.702016:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829113.702029:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3663 00000100:00080000:0.0:1787829113.702042:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829113.702052:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829113.702063:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829113.702113:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280846080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829113.702137:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829113.702140:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829113.702141:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829113.702144:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3663 00000100:00080000:0.0:1787829113.702147:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829113.702151:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829113.702162:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829113.702175:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280846208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829113.702181:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280846080:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829113.702190:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829113.702194:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829113.702196:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829113.702200:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3663 00000100:00080000:0.0:1787829113.702204:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829113.702209:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829113.702214:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829113.702228:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280846336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829113.702246:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829113.702251:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829113.702253:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829113.702255:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829113.702268:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3663 00000100:00080000:0.0:1787829113.702275:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829113.702282:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829113.702290:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829113.702306:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829113/real 1787829113] req@ffff8ab3f075e580 x1874671280846080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829113.702311:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280846464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829113.702339:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280846080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829113.702341:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829113.702360:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829113.702367:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829113.702367:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829113.702381:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787829113.702386:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3663 00000100:00100000:1.0:1787829113.702389:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280846080:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829113.702397:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829113.702406:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829113.702414:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829113.702416:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280846208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787829113.702433:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280846592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829113.702446:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829113.702479:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829113/real 1787829113] req@ffff8ab3f075f480 x1874671280846208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829113.702495:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280846208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829113.702505:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829113.702509:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829113.702513:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280846208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829113.702519:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280846336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829113.702536:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829113.702544:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829113/real 1787829113] req@ffff8ab3f075da40 x1874671280846336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829113.702552:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280846336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829113.702557:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829113.702559:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829113.702561:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280846336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829113.702565:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280846464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829113.702576:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829113.702581:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829113/real 1787829113] req@ffff8ab3f075ed00 x1874671280846464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829113.702588:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280846464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829113.702595:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829113.702597:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829113.702599:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280846464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829113.702612:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280846592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829113.702621:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829113.702626:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829113/real 1787829113] req@ffff8ab3f075e940 x1874671280846592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829113.702633:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280846592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829113.702639:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829113.702641:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829113.702643:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280846592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829114.726044:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829115.750055:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829116.774043:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829117.798049:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829118.822031:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829118.822043:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829118.822049:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829118.822060:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3668 00000100:00080000:3.0:1787829118.822072:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829118.822080:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829118.822101:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829118.822133:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280846720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829118.822156:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829118.822159:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829118.822161:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829118.822164:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3668 00000100:00080000:3.0:1787829118.822166:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829118.822168:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829118.822178:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829118.822188:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280846848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829118.822197:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829118.822200:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829118.822202:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829118.822208:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3668 00000100:00100000:1.0:1787829118.822209:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280846720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829118.822212:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829118.822223:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829118.822227:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829118.822242:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280846976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829118.822253:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829118.822256:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829118.822257:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829118.822260:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3668 00000100:00080000:3.0:1787829118.822277:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829118.822281:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829118.822284:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829118.822291:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829118.822297:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280847104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829118.822307:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829118.822310:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829118.822312:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829118.822318:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3668 00000100:00080000:3.0:1787829118.822321:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829118.822323:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829118.822327:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829118.822338:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280847232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829118.822367:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829118/real 1787829118] req@ffff8ab3d0a41e00 x1874671280846720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829118.822383:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280846720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829118.822392:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829118.822396:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829118.822399:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280846720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829118.822423:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280846848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829118.822449:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829118.822464:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829118/real 1787829118] req@ffff8ab3d0a430c0 x1874671280846848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829118.822478:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280846848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829118.822484:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829118.822487:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829118.822489:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280846848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829118.822494:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280846976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829118.822516:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829118.822537:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829118/real 1787829118] req@ffff8ab3d0a43c00 x1874671280846976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829118.822550:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280846976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829118.822559:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829118.822563:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829118.822566:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280846976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829118.822573:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280847104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829118.822597:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829118.822611:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829118/real 1787829118] req@ffff8ab3d0a43480 x1874671280847104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829118.822624:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280847104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829118.822635:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829118.822639:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829118.822641:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280847104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829118.822646:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280847232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829118.822663:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829118.822671:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829118/real 1787829118] req@ffff8ab3d0a412c0 x1874671280847232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829118.822679:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280847232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829118.822685:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829118.822687:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829118.822690:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280847232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829119.846089:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829120.870091:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829121.894047:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829122.918063:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829123.942015:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829123.942026:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829123.942031:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829123.942044:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3673 00000100:00080000:0.0:1787829123.942055:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829123.942065:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829123.942095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829123.942109:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280847360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829123.942157:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829123.942160:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829123.942162:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829123.942165:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3673 00000100:00080000:0.0:1787829123.942168:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829123.942170:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829123.942178:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829123.942187:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280847488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829123.942195:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829123.942197:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829123.942199:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829123.942203:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3673 00000100:00080000:0.0:1787829123.942205:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829123.942207:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829123.942211:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829123.942219:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280847360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829123.942221:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280847616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829123.942233:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829123.942236:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829123.942237:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829123.942240:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3673 00000100:00080000:0.0:1787829123.942242:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829123.942245:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829123.942277:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829123.942293:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280847744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829123.942303:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829123.942307:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829123.942308:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829123.942318:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3673 00000800:00000100:1.0:1787829123.942320:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829123.942325:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829123.942331:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829123.942338:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829123.942352:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280847872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829123.942395:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829123/real 1787829123] req@ffff8ab3f075f840 x1874671280847360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829123.942419:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280847360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829123.942451:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829123.942455:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829123.942457:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280847360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829123.942463:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280847488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829123.942480:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829123.942489:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829123/real 1787829123] req@ffff8ab3f075cb40 x1874671280847488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829123.942497:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280847488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829123.942503:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829123.942505:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829123.942507:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280847488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829123.942530:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280847616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829123.942550:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829123.942562:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829123/real 1787829123] req@ffff8ab3f075f0c0 x1874671280847616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829123.942575:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280847616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829123.942603:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829123.942607:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829123.942611:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280847616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829123.942622:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280847744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829123.942655:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829123.942674:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829123/real 1787829123] req@ffff8ab3f075de00 x1874671280847744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829123.942695:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280847744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829123.942711:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829123.942716:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829123.942720:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280847744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829123.942727:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280847872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829123.942750:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829123.942786:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829123/real 1787829123] req@ffff8ab3f075cf00 x1874671280847872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829123.942812:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280847872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829123.942819:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829123.942821:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829123.942823:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280847872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829124.966042:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829125.990061:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829127.014081:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829128.038082:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829129.062032:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829129.062043:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829129.062048:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829129.062059:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3678 00010000:00010000:2.0:1787829129.062063:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829129.062071:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829129.062080:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829129.062124:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280848000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829129.062150:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829129.062153:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829129.062154:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829129.062157:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3678 00000100:00080000:3.0:1787829129.062159:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829129.062168:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829129.062178:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829129.062193:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280848000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829129.062193:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280848128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829129.062217:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829129.062220:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829129.062221:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829129.062227:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3678 00000100:00080000:3.0:1787829129.062232:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829129.062237:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829129.062243:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829129.062267:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280848256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829129.062279:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829129.062282:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829129.062287:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829129.062288:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829129.062295:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3678 00000100:00080000:3.0:1787829129.062301:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829129.062306:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829129.062312:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829129.062327:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280848384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829129.062337:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829129/real 1787829129] req@ffff8ab3d0a43840 x1874671280848000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829129.062339:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829129.062353:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829129.062360:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829129.062375:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3678 00000100:00080000:3.0:1787829129.062380:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829129.062385:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280848000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829129.062390:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829129.062399:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829129.062416:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280848512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829129.062420:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829129.062431:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829129.062437:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280848000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829129.062468:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280848128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829129.062489:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829129.062500:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829129/real 1787829129] req@ffff8ab3d0a40000 x1874671280848128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829129.062509:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280848128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829129.062515:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829129.062518:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829129.062520:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280848128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829129.062525:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280848256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829129.062535:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829129.062541:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829129/real 1787829129] req@ffff8ab3d0a421c0 x1874671280848256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829129.062549:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280848256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829129.062559:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829129.062561:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829129.062563:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280848256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829129.062567:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280848384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829129.062578:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829129.062584:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829129/real 1787829129] req@ffff8ab3d0a412c0 x1874671280848384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829129.062591:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280848384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829129.062597:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829129.062600:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829129.062602:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280848384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829129.062606:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280848512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829129.062615:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829129.062621:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829129/real 1787829129] req@ffff8ab3d0a43480 x1874671280848512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829129.062627:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280848512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829129.062633:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829129.062635:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829129.062638:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280848512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829130.086075:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829131.110166:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829132.134051:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829133.158063:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829134.182008:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829134.182018:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829134.182022:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829134.182033:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3683 00010000:00010000:3.0:1787829134.182040:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829134.182044:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829134.182054:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829134.182088:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280848640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829134.182114:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829134.182118:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829134.182120:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829134.182126:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3683 00000100:00080000:0.0:1787829134.182130:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829134.182135:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829134.182145:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829134.182148:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280848640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829134.182163:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280848768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829134.182177:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829134.182182:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829134.182184:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829134.182189:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3683 00000100:00080000:0.0:1787829134.182203:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829134.182213:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829134.182218:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829134.182225:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829134.182242:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280848896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829134.182253:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829134.182257:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829134/real 1787829134] req@ffff8ab3f075c000 x1874671280848640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829134.182259:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829134.182264:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829134.182274:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3683 00000100:00080000:0.0:1787829134.182281:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829134.182281:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280848640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829134.182292:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829134.182297:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787829134.182301:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829134.182303:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280848640:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787829134.182312:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829134.182329:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280849024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829134.182333:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280848768:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787829134.182349:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829134.182353:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829134.182356:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829134.182362:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3683 00000100:00080000:0.0:1787829134.182367:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829134.182368:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829134.182377:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829134.182384:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829134.182386:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829134/real 1787829134] req@ffff8ab3f075cf00 x1874671280848768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829134.182403:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280849152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829134.182404:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280848768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829134.182429:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829134.182436:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829134.182441:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280848768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829134.182450:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280848896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829134.182472:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829134.182494:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829134/real 1787829134] req@ffff8ab3f075de00 x1874671280848896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829134.182508:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280848896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829134.182517:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829134.182521:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829134.182524:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280848896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829134.182536:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280849024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829134.182558:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829134.182566:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829134/real 1787829134] req@ffff8ab3f075f0c0 x1874671280849024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829134.182574:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280849024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829134.182581:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829134.182583:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829134.182585:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280849024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829134.182589:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280849152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829134.182598:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829134.182603:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829134/real 1787829134] req@ffff8ab3f075cb40 x1874671280849152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829134.182610:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280849152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829134.182616:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829134.182618:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829134.182620:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280849152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829135.206058:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829136.230057:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829137.254042:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829138.278239:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829139.301995:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829139.302007:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829139.302013:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1787829139.302021:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829139.302027:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3688 00000100:00080000:3.0:1787829139.302042:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829139.302050:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829139.302098:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280849280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829139.302129:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829139.302133:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829139.302135:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829139.302139:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3688 00000100:00080000:3.0:1787829139.302143:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829139.302148:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829139.302160:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829139.302173:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280849280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829139.302177:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280849408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829139.302189:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829139.302193:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829139.302195:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829139.302199:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3688 00000100:00080000:3.0:1787829139.302202:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829139.302206:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829139.302211:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829139.302223:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280849536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829139.302234:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829139.302237:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829139.302239:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829139.302244:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3688 00000800:00000100:1.0:1787829139.302249:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829139.302251:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829139.302258:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829139.302264:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829139.302277:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280849664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829139.302286:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829139.302290:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829139.302291:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829139.302296:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3688 00000100:00080000:3.0:1787829139.302300:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829139.302304:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829139.302310:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829139.302315:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829139/real 1787829139] req@ffff8ab3d0a43c00 x1874671280849280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829139.302325:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280849792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829139.302339:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280849280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829139.302356:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829139.302362:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829139.302367:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280849280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829139.302394:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280849408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829139.302414:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829139.302424:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829139/real 1787829139] req@ffff8ab3d0a430c0 x1874671280849408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829139.302433:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280849408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829139.302440:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829139.302443:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829139.302447:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280849408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829139.302454:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280849536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829139.302474:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829139.302487:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829139/real 1787829139] req@ffff8ab3d0a41e00 x1874671280849536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829139.302499:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280849536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829139.302506:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829139.302508:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829139.302510:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280849536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829139.302515:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280849664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829139.302530:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829139.302537:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829139/real 1787829139] req@ffff8ab3d0a43480 x1874671280849664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829139.302544:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280849664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829139.302551:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829139.302553:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829139.302555:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280849664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829139.302558:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280849792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829139.302569:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829139.302575:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829139/real 1787829139] req@ffff8ab3d0a412c0 x1874671280849792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829139.302583:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280849792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829139.302594:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829139.302597:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829139.302601:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280849792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829140.326087:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829141.350054:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829142.374030:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829143.398026:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829144.421994:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829144.422008:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829144.422013:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829144.422026:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3694 00000100:00080000:3.0:1787829144.422040:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829144.422050:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829144.422062:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829144.422099:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280849920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829144.422124:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829144.422126:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829144.422128:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829144.422130:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3694 00000100:00080000:3.0:1787829144.422133:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829144.422137:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829144.422147:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829144.422157:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280850048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829144.422163:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280849920:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829144.422169:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829144.422172:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829144.422174:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829144.422177:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3694 00000100:00080000:3.0:1787829144.422179:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829144.422182:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829144.422185:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829144.422195:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280850176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829144.422202:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829144.422204:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829144.422205:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829144.422208:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3694 00000100:00080000:3.0:1787829144.422230:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829144.422233:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829144.422236:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829144.422245:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829144.422248:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280850304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829144.422256:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829144.422259:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829144.422261:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829144.422264:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3694 00000100:00080000:3.0:1787829144.422266:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829144.422269:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829144.422272:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829144.422279:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280850432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829144.422312:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829144/real 1787829144] req@ffff8ab3d0a421c0 x1874671280849920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829144.422332:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280849920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829144.422342:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829144.422345:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829144.422348:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280849920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829144.422371:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280850048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829144.422390:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829144.422400:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829144/real 1787829144] req@ffff8ab3d0a40000 x1874671280850048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829144.422408:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280850048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829144.422419:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829144.422422:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829144.422424:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280850048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829144.422428:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280850176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829144.422438:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829144.422444:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829144/real 1787829144] req@ffff8ab3d0a43840 x1874671280850176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829144.422451:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280850176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829144.422456:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829144.422458:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829144.422460:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280850176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829144.422463:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280850304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829144.422475:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829144.422480:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829144/real 1787829144] req@ffff8ab3d0a412c0 x1874671280850304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829144.422488:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280850304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829144.422494:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829144.422496:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829144.422498:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280850304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829144.422502:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280850432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829144.422513:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829144.422518:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829144/real 1787829144] req@ffff8ab3d0a43480 x1874671280850432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829144.422525:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280850432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829144.422531:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829144.422533:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829144.422535:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280850432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829145.446021:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829146.470069:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829147.494064:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829148.518067:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829149.542042:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829149.542052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829149.542057:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829149.542069:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3699 00010000:00010000:3.0:1787829149.542075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829149.542081:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829149.542090:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829149.542136:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280850560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829149.542164:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829149.542167:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829149.542169:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829149.542171:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3699 00000100:00080000:2.0:1787829149.542174:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829149.542177:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829149.542186:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829149.542198:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280850688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829149.542205:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829149.542208:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829149.542209:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829149.542215:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3699 00000100:00100000:1.0:1787829149.542216:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280850560:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829149.542217:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829149.542224:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829149.542231:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829149.542245:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280850816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829149.542256:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829149.542260:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829149.542262:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829149.542267:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3699 00000100:00080000:2.0:1787829149.542291:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829149.542296:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829149.542302:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829149.542316:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280850944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829149.542326:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829149.542329:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829149.542330:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829149.542334:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3699 00000100:00080000:2.0:1787829149.542338:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829149.542341:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829149.542345:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829149.542354:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280851072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829149.542370:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280850688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829149.542411:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280850816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829149.542449:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280850944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829149.542478:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280851072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829150.566128:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829151.590061:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829152.614204:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787829152.614268:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:3.0:1787829152.614313:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829152.614338:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787829152.614395:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787829152.614406:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787829152.614415:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787829152.614425:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787829152.614465:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829149/real 1787829152] req@ffff8ab3f0438780 x1874671280850688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829152.614499:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280850688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829152.614521:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829152.614528:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829152.614544:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280850688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829152.614567:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829149/real 1787829152] req@ffff8ab3f0438b40 x1874671280850816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829152.614583:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280850816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829152.614591:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829152.614595:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829152.614599:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280850816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829152.614607:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829149/real 1787829152] req@ffff8ab3f0438f00 x1874671280850944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829152.614620:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280850944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829152.614632:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829152.614635:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829152.614640:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280850944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829152.614648:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829149/real 1787829152] req@ffff8ab3f043a580 x1874671280851072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829152.614661:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280851072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829152.614677:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829152.614684:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829152.614688:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280851072:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829152.614696:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829149/real 1787829152] req@ffff8ab3f043b0c0 x1874671280850560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829152.614709:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280850560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829152.614718:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829152.614721:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829152.614724:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280850560:192.168.203.124@tcp:250:kworker.0 00010000:00010000:3.0:1787829153.638043:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829154.662041:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829154.662051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829154.662055:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829154.662066:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3704 00010000:00010000:3.0:1787829154.662073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829154.662076:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829154.662090:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829154.662130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280851200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829154.662162:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829154.662167:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829154.662169:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829154.662173:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3704 00000100:00080000:0.0:1787829154.662177:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829154.662182:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829154.662193:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829154.662207:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280851328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829154.662217:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829154.662218:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280851200:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829154.662221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829154.662223:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829154.662227:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3704 00000100:00080000:0.0:1787829154.662230:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829154.662234:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829154.662238:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829154.662248:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280851456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829154.662257:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829154.662260:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829154.662262:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829154.662266:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3704 00000100:00080000:0.0:1787829154.662289:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829154.662293:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829154.662298:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829154.662302:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829154.662308:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280851584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829154.662316:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829154.662319:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829154.662321:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829154.662325:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3704 00000100:00080000:0.0:1787829154.662337:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829154.662342:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829154.662347:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829154.662357:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280851712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829154.662358:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829154/real 1787829154] req@ffff8ab3f075f840 x1874671280851200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829154.662385:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280851200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829154.662398:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829154.662403:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829154.662408:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280851200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829154.662448:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280851328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829154.662476:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829154.662491:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829154/real 1787829154] req@ffff8ab3f075e940 x1874671280851328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829154.662503:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280851328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829154.662511:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829154.662515:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829154.662518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280851328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829154.662557:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280851456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829154.662581:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829154.662609:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829154/real 1787829154] req@ffff8ab3f075ed00 x1874671280851456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829154.662621:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280851456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829154.662630:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829154.662633:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829154.662636:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280851456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829154.662642:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280851584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829154.662663:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829154.662675:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829154/real 1787829154] req@ffff8ab3f075da40 x1874671280851584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829154.662695:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280851584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829154.662705:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829154.662709:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829154.662712:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280851584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829154.662718:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280851712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829154.662739:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829154.662751:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829154/real 1787829154] req@ffff8ab3f075f480 x1874671280851712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829154.662763:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280851712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829154.662774:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829154.662778:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829154.662781:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280851712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829155.686106:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829156.710089:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829157.734101:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829158.758069:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829159.782106:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829159.782115:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829159.782118:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829159.782127:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3709 00000100:00080000:3.0:1787829159.782136:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829159.782144:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829159.782163:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829159.782181:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280851840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829159.782216:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829159.782220:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829159.782222:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829159.782227:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3709 00000100:00080000:3.0:1787829159.782231:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829159.782235:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829159.782246:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829159.782260:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280851968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829159.782269:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280851840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829159.782271:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829159.782277:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829159.782279:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829159.782282:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3709 00000100:00080000:3.0:1787829159.782285:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829159.782288:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829159.782292:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829159.782301:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280852096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829159.782308:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829159.782311:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829159.782312:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829159.782314:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3709 00000100:00080000:3.0:1787829159.782325:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829159.782328:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829159.782333:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787829159.782336:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829159.782348:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280852224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829159.782354:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829159.782356:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829159.782358:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829159.782361:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3709 00000100:00080000:3.0:1787829159.782363:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829159.782366:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829159.782369:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829159.782378:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280852352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829159.782404:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829159/real 1787829159] req@ffff8ab3d0a41e00 x1874671280851840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829159.782424:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280851840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829159.782437:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829159.782440:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829159.782444:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280851840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829159.782464:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280851968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829159.782483:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829159.782497:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829159/real 1787829159] req@ffff8ab3d0a430c0 x1874671280851968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829159.782525:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280851968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829159.782534:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829159.782538:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829159.782541:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280851968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829159.782555:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280852096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829159.782577:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829159.782589:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829159/real 1787829159] req@ffff8ab3d0a43c00 x1874671280852096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829159.782601:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280852096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829159.782607:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829159.782609:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829159.782612:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280852096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829159.782616:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280852224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829159.782631:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829159.782638:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829159/real 1787829159] req@ffff8ab3d0a43480 x1874671280852224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829159.782645:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280852224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829159.782651:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829159.782653:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829159.782655:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280852224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829159.782658:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280852352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829159.782667:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829159.782673:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829159/real 1787829159] req@ffff8ab3d0a412c0 x1874671280852352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829159.782680:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280852352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829159.782686:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829159.782687:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829159.782689:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280852352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829160.806048:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829161.830169:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829162.854140:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829163.878095:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829164.902021:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829164.902032:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829164.902038:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829164.902048:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3714 00000100:00080000:0.0:1787829164.902061:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829164.902070:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829164.902081:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829164.902122:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280852480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829164.902151:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829164.902156:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829164.902158:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829164.902163:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3714 00000100:00080000:0.0:1787829164.902168:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829164.902173:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829164.902184:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829164.902202:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280852608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829164.902202:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280852480:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829164.902219:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829164.902224:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829164.902227:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829164.902232:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3714 00000100:00080000:0.0:1787829164.902237:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829164.902243:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829164.902249:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829164.902264:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829164.902277:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280852736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829164.902289:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829164.902293:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829164.902296:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829164.902301:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3714 00000100:00080000:0.0:1787829164.902306:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829164.902311:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829164.902314:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829164/real 1787829164] req@ffff8ab3f075e580 x1874671280852480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829164.902321:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829164.902334:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280852480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829164.902340:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280852864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829164.902345:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829164.902352:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829164.902358:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280852480:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829164.902361:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829164.902367:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829164.902369:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829164.902376:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3714 00000100:00100000:1.0:1787829164.902380:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280852608:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787829164.902381:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829164.902390:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829164.902397:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787829164.902401:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829164.902413:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829164/real 1787829164] req@ffff8ab3f075f480 x1874671280852608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829164.902418:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280852992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829164.902430:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280852608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829164.902441:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829164.902446:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829164.902449:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280852608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829164.902453:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280852736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829164.902466:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829164.902491:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829164/real 1787829164] req@ffff8ab3f075da40 x1874671280852736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829164.902500:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280852736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829164.902510:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829164.902513:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829164.902515:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280852736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829164.902519:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280852864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829164.902534:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829164.902540:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829164/real 1787829164] req@ffff8ab3f075ed00 x1874671280852864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829164.902547:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280852864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829164.902554:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829164.902556:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829164.902557:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280852864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829164.902568:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280852992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829164.902579:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829164.902585:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829164/real 1787829164] req@ffff8ab3f075e940 x1874671280852992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829164.902592:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280852992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829164.902598:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829164.902600:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829164.902603:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280852992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829165.926102:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829166.950031:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829167.974094:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829168.998100:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829170.021988:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829170.021998:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829170.022002:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829170.022011:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3719 00000100:00080000:0.0:1787829170.022019:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829170.022026:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829170.022051:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829170.022060:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280853120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829170.022084:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829170.022087:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829170.022088:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829170.022091:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3719 00000100:00080000:0.0:1787829170.022095:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829170.022098:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829170.022106:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829170.022113:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280853120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829170.022117:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280853248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829170.022127:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829170.022130:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829170.022131:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829170.022134:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3719 00000100:00080000:0.0:1787829170.022136:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829170.022139:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829170.022141:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829170.022150:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280853376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829170.022155:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829170.022158:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829170.022160:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829170.022163:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3719 00000100:00080000:0.0:1787829170.022177:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829170.022182:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829170.022187:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829170.022189:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829170.022203:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280853504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829170.022222:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829170.022226:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829170.022228:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829170.022234:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3719 00000100:00080000:0.0:1787829170.022238:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829170.022241:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829170.022246:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829170/real 1787829170] req@ffff8ab3f075f840 x1874671280853120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829170.022247:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829170.022261:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280853632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829170.022271:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280853120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829170.022291:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829170.022296:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829170.022301:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280853120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829170.022324:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280853248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829170.022347:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829170.022361:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829170/real 1787829170] req@ffff8ab3f075cb40 x1874671280853248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829170.022370:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280853248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829170.022376:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829170.022378:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829170.022381:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280853248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829170.022385:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280853376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829170.022398:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829170.022405:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829170/real 1787829170] req@ffff8ab3f075f0c0 x1874671280853376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829170.022412:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280853376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829170.022418:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829170.022419:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829170.022421:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280853376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829170.022425:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280853504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829170.022452:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829170.022460:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829170/real 1787829170] req@ffff8ab3f075de00 x1874671280853504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829170.022468:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280853504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829170.022475:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829170.022477:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829170.022480:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280853504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829170.022484:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280853632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829170.022495:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829170.022500:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829170/real 1787829170] req@ffff8ab3f075cf00 x1874671280853632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829170.022508:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280853632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829170.022514:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829170.022516:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829170.022518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280853632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829171.046105:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829172.070181:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829173.094081:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829174.118097:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829175.142083:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829175.142100:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829175.142106:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829175.142134:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3724 00000100:00080000:0.0:1787829175.142165:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829175.142176:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829175.142180:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829175.142242:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280853760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829175.142277:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829175.142282:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829175.142284:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829175.142289:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3724 00000100:00080000:0.0:1787829175.142294:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829175.142300:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829175.142331:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829175.142336:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280853760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829175.142356:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280853888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829175.142374:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829175.142380:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829175.142383:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829175.142389:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3724 00000100:00080000:0.0:1787829175.142433:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829175.142440:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829175.142448:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787829175.142449:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829175.142473:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280854016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829175.142499:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829175.142504:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829175.142507:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829175.142513:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3724 00000100:00080000:0.0:1787829175.142520:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829175.142523:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829175/real 1787829175] req@ffff8ab3f075c000 x1874671280853760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829175.142531:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829175.142540:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829175.142580:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280854144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829175.142591:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280853760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829175.142625:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829175.142626:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829175.142634:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829175.142636:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787829175.142639:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829175.142657:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280853760:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829175.142661:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3724 00000100:00080000:0.0:1787829175.142670:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829175.142677:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829175.142684:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829175.142717:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280853888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787829175.142720:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280854272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829175.142760:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829175.142776:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829175/real 1787829175] req@ffff8ab3f075cf00 x1874671280853888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829175.142788:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280853888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829175.142797:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829175.142800:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829175.142804:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280853888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829175.142809:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280854016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829175.142849:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829175.142857:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829175/real 1787829175] req@ffff8ab3f075de00 x1874671280854016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829175.142866:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280854016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829175.142872:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829175.142874:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829175.142876:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280854016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829175.142880:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280854144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829175.142892:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829175.142916:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829175/real 1787829175] req@ffff8ab3f075f0c0 x1874671280854144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829175.142935:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280854144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829175.142943:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829175.142980:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829175.142983:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280854144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829175.142997:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280854272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829175.143013:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829175.143024:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829175/real 1787829175] req@ffff8ab3f075cb40 x1874671280854272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829175.143039:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280854272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829175.143067:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829175.143071:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829175.143075:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280854272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829176.166033:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829177.190139:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829178.214023:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829179.238076:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829180.261997:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829180.262008:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829180.262012:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829180.262023:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3729 00000100:00080000:0.0:1787829180.262033:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829180.262039:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829180.262074:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280854400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787829180.262081:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829180.262108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829180.262114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829180.262117:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829180.262123:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3729 00000100:00080000:0.0:1787829180.262128:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829180.262133:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829180.262142:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829180.262148:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280854400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829180.262163:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280854528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829180.262186:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829180.262190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829180.262193:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829180.262200:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3729 00000100:00080000:0.0:1787829180.262221:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829180.262229:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829180.262230:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787829180.262243:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829180.262259:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280854656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829180.262267:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829180.262271:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829180.262273:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829180.262276:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3729 00000100:00080000:0.0:1787829180.262280:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829180.262284:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829180.262288:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829180.262293:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829180/real 1787829180] req@ffff8ab3f075f840 x1874671280854400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829180.262298:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280854784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829180.262341:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829180.262345:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829180.262349:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829180.262349:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280854400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829180.262376:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3729 00000100:00080000:0.0:1787829180.262387:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829180.262396:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829180.262401:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787829180.262403:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829180.262415:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787829180.262416:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280854912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829180.262434:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280854400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829180.262471:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280854528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829180.262497:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829180.262512:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829180/real 1787829180] req@ffff8ab3f075e940 x1874671280854528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829180.262527:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280854528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829180.262543:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829180.262547:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829180.262550:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280854528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829180.262557:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280854656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829180.262578:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829180.262587:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829180/real 1787829180] req@ffff8ab3f075ed00 x1874671280854656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829180.262595:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280854656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829180.262601:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829180.262604:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829180.262606:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280854656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829180.262610:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280854784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829180.262622:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829180.262628:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829180/real 1787829180] req@ffff8ab3f075da40 x1874671280854784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829180.262636:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280854784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829180.262642:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829180.262644:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829180.262647:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280854784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829180.262651:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280854912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829180.262662:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829180.262667:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829180/real 1787829180] req@ffff8ab3f075f480 x1874671280854912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829180.262675:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280854912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829180.262681:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829180.262683:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829180.262685:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280854912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829181.286055:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829182.310024:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829183.334103:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829184.358088:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829185.382041:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829185.382052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829185.382057:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829185.382070:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3735 00000100:00080000:0.0:1787829185.382082:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829185.382090:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829185.382113:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829185.382143:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280855040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829185.382169:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829185.382171:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829185.382173:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829185.382175:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3735 00000100:00080000:0.0:1787829185.382178:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829185.382181:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829185.382191:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829185.382202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280855168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829185.382214:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280855040:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829185.382216:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829185.382234:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829185.382236:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829185.382240:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3735 00000100:00080000:0.0:1787829185.382242:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829185.382246:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829185.382250:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829185.382261:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280855296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829185.382268:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829185.382271:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829185.382272:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829185.382275:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3735 00000100:00080000:0.0:1787829185.382284:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829185.382288:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829185.382294:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787829185.382295:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829185.382310:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280855424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829185.382318:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829185.382321:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829185.382322:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829185.382339:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3735 00000100:00080000:0.0:1787829185.382344:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829185.382348:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829185.382355:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829185.382360:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829185/real 1787829185] req@ffff8ab3f075e580 x1874671280855040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829185.382369:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280855552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829185.382392:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280855040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829185.382402:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829185.382406:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829185.382409:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280855040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829185.382430:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280855168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829185.382448:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829185.382458:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829185/real 1787829185] req@ffff8ab3f075f480 x1874671280855168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829185.382466:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280855168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829185.382472:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829185.382474:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829185.382476:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280855168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829185.382480:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280855296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829185.382491:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829185.382497:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829185/real 1787829185] req@ffff8ab3f075da40 x1874671280855296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829185.382509:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280855296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829185.382518:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829185.382522:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829185.382525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280855296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829185.382533:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280855424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829185.382555:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829185.382568:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829185/real 1787829185] req@ffff8ab3f075ed00 x1874671280855424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829185.382581:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280855424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829185.382592:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829185.382596:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829185.382600:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280855424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829185.382607:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280855552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829185.382629:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829185.382642:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829185/real 1787829185] req@ffff8ab3f075e940 x1874671280855552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829185.382654:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280855552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829185.382664:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829185.382668:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829185.382672:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280855552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829186.406055:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829187.430085:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829188.454033:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829189.478045:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829190.502034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829190.502044:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829190.502049:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829190.502061:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3740 00000100:00080000:0.0:1787829190.502072:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787829190.502075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829190.502081:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829190.502130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280855680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829190.502152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829190.502154:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829190.502156:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829190.502158:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3740 00000100:00080000:0.0:1787829190.502161:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829190.502164:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829190.502174:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829190.502184:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280855808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829190.502190:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280855680:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829190.502202:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829190.502211:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829190.502213:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829190.502217:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3740 00000100:00080000:0.0:1787829190.502219:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829190.502222:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829190.502226:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829190.502236:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280855936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829190.502242:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829190.502245:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829190.502246:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829190.502249:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3740 00000100:00080000:0.0:1787829190.502270:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829190.502273:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829190.502277:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829190.502282:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829190.502287:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280856064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829190.502309:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829190.502312:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829190.502313:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829190.502317:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3740 00000100:00080000:0.0:1787829190.502323:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829190.502326:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829190.502331:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829190.502353:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280856192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829190.502369:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829190/real 1787829190] req@ffff8ab3f075f840 x1874671280855680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829190.502386:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280855680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829190.502395:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829190.502398:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829190.502412:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280855680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829190.502437:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280855808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829190.502454:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829190.502464:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829190/real 1787829190] req@ffff8ab3f075cb40 x1874671280855808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829190.502472:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280855808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829190.502489:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829190.502491:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829190.502494:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280855808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829190.502497:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280855936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829190.502508:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829190.502514:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829190/real 1787829190] req@ffff8ab3f075f0c0 x1874671280855936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829190.502521:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280855936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829190.502527:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829190.502528:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829190.502531:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280855936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829190.502536:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280856064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829190.502567:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829190.502580:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829190/real 1787829190] req@ffff8ab3f075de00 x1874671280856064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829190.502594:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280856064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829190.502606:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829190.502609:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829190.502613:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280856064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829190.502618:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280856192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829190.502644:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829190.502651:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829190/real 1787829190] req@ffff8ab3f075cf00 x1874671280856192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829190.502659:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280856192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829190.502667:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829190.502669:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829190.502671:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280856192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829191.526144:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829192.550095:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829193.574143:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829194.598138:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829195.622031:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829195.622043:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829195.622047:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829195.622059:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3745 00000100:00080000:3.0:1787829195.622070:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1787829195.622085:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829195.622085:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829195.622153:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280856320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829195.622189:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829195.622194:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829195.622196:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829195.622201:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3745 00000100:00080000:3.0:1787829195.622205:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829195.622210:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829195.622225:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829195.622240:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280856448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829195.622302:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829195.622305:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280856320:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829195.622308:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829195.622311:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829195.622327:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3745 00000100:00080000:3.0:1787829195.622333:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829195.622353:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829195.622358:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829195.622372:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280856576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829195.622380:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829195.622384:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829195.622387:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829195.622393:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3745 00000100:00080000:3.0:1787829195.622421:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829195.622428:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829195.622438:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787829195.622439:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829195.622467:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280856704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829195.622479:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829195.622484:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829195.622486:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829195.622494:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3745 00000100:00080000:3.0:1787829195.622516:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829195.622537:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829195.622547:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829195.622548:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829195/real 1787829195] req@ffff8ab3d0a43840 x1874671280856320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829195.622574:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280856832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829195.622598:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280856320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829195.622627:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829195.622635:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829195.622641:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280856320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829195.622689:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280856448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829195.622713:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829195.622735:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829195/real 1787829195] req@ffff8ab3d0a40000 x1874671280856448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829195.622744:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280856448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829195.622751:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829195.622754:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829195.622756:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280856448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829195.622761:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280856576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829195.622773:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829195.622779:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829195/real 1787829195] req@ffff8ab3d0a421c0 x1874671280856576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829195.622798:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280856576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829195.622804:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829195.622806:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829195.622809:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280856576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829195.622813:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280856704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829195.622824:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829195.622831:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829195/real 1787829195] req@ffff8ab3d0a412c0 x1874671280856704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829195.622838:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280856704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829195.622845:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829195.622847:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829195.622849:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280856704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829195.622863:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280856832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829195.622873:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829195.622879:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829195/real 1787829195] req@ffff8ab3d0a43480 x1874671280856832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829195.622912:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280856832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829195.622930:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829195.622933:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829195.622935:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280856832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829196.646187:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829197.670120:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829198.694093:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829199.718073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829200.742014:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829200.742026:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829200.742031:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829200.742042:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3750 00000100:00080000:3.0:1787829200.742054:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829200.742061:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829200.742107:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280856960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829200.742136:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829200.742140:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829200.742142:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829200.742145:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3750 00000100:00080000:3.0:1787829200.742147:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829200.742153:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829200.742160:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:3.0:1787829200.742164:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829200.742176:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280856960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829200.742177:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280857088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829200.742192:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829200.742194:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829200.742196:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829200.742199:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3750 00000100:00080000:3.0:1787829200.742201:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829200.742204:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829200.742207:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829200.742216:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280857216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829200.742238:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829200.742241:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829200.742243:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829200.742246:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3750 00000100:00080000:3.0:1787829200.742256:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829200.742260:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829200.742265:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829200.742267:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829200.742280:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280857344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829200.742289:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829200.742293:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829200.742295:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829200.742300:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3750 00000100:00080000:3.0:1787829200.742305:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829200.742310:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829200.742317:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829200.742326:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829200/real 1787829200] req@ffff8ab3d0a43c00 x1874671280856960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829200.742330:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280857472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829200.742363:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280856960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829200.742373:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829200.742377:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829200.742380:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280856960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829200.742402:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280857088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829200.742417:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829200.742426:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829200/real 1787829200] req@ffff8ab3d0a430c0 x1874671280857088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829200.742434:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280857088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829200.742440:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829200.742441:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829200.742443:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280857088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829200.742447:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280857216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829200.742458:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829200.742463:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829200/real 1787829200] req@ffff8ab3d0a41e00 x1874671280857216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829200.742471:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280857216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829200.742481:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829200.742483:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829200.742486:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280857216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829200.742489:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280857344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829200.742499:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829200.742504:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829200/real 1787829200] req@ffff8ab3d0a43480 x1874671280857344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829200.742511:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280857344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829200.742519:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829200.742520:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829200.742522:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280857344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829200.742526:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280857472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829200.742534:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829200.742540:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829200/real 1787829200] req@ffff8ab3d0a412c0 x1874671280857472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829200.742547:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280857472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829200.742553:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829200.742555:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829200.742558:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280857472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829201.766095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829202.790069:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829203.814103:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829204.838038:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829205.862035:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1787829205.862035:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829205.862046:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829205.862051:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829205.862063:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3755 00000100:00080000:0.0:1787829205.862074:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829205.862093:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829205.862127:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280857600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829205.862161:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829205.862164:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829205.862166:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829205.862169:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3755 00000100:00080000:0.0:1787829205.862172:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829205.862176:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829205.862196:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829205.862211:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280857728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829205.862221:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829205.862225:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829205.862239:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829205.862242:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280857600:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829205.862249:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3755 00000100:00080000:0.0:1787829205.862259:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829205.862266:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829205.862273:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829205.862289:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280857856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829205.862301:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829205.862314:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829205.862315:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829205.862319:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3755 00000100:00080000:0.0:1787829205.862343:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829205.862347:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829205.862352:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829205.862358:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829205.862363:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280857984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829205.862372:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829205.862374:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829205.862376:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829205.862381:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3755 00000100:00080000:0.0:1787829205.862385:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829205.862389:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829205.862396:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829205.862414:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280858112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829205.862431:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829205/real 1787829205] req@ffff8ab3f075c000 x1874671280857600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829205.862454:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280857600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829205.862464:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829205.862469:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829205.862473:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280857600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829205.862515:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280857728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829205.862537:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829205.862552:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829205/real 1787829205] req@ffff8ab3f075cf00 x1874671280857728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829205.862565:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280857728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829205.862575:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829205.862579:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829205.862583:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280857728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829205.862598:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280857856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829205.862615:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829205.862625:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829205/real 1787829205] req@ffff8ab3f075de00 x1874671280857856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829205.862633:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280857856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829205.862639:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829205.862642:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829205.862644:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280857856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829205.862648:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280857984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829205.862660:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829205.862666:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829205/real 1787829205] req@ffff8ab3f075f0c0 x1874671280857984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829205.862673:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280857984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829205.862681:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829205.862683:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829205.862684:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280857984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829205.862688:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280858112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829205.862699:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829205.862704:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829205/real 1787829205] req@ffff8ab3f075cb40 x1874671280858112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829205.862711:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280858112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829205.862719:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829205.862721:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829205.862723:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280858112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829206.886037:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829207.910136:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829208.934075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829209.958067:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829210.982023:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829210.982034:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829210.982039:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829210.982050:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3760 00000100:00080000:2.0:1787829210.982060:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829210.982067:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829210.982091:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829210.982114:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280858240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829210.982136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829210.982138:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829210.982140:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829210.982142:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3760 00000100:00080000:2.0:1787829210.982145:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829210.982161:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829210.982172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829210.982181:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280858240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829210.982186:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280858368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829210.982195:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829210.982197:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829210.982199:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829210.982202:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3760 00000100:00080000:2.0:1787829210.982205:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829210.982208:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829210.982211:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829210.982222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280858496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829210.982228:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829210.982231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829210.982232:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829210.982235:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3760 00000100:00080000:2.0:1787829210.982247:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829210.982250:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829210.982254:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829210.982258:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829210.982265:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280858624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829210.982273:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829210.982275:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829210.982277:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829210.982281:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3760 00000100:00080000:2.0:1787829210.982283:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829210.982285:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829210.982288:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829210.982296:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280858752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829210.982320:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829210/real 1787829210] req@ffff8ab3f0439680 x1874671280858240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829210.982343:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280858240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829210.982354:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829210.982357:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829210.982360:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280858240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829210.982382:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280858368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829210.982400:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829210.982410:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829210/real 1787829210] req@ffff8ab3f0439a40 x1874671280858368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829210.982418:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280858368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829210.982424:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829210.982427:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829210.982429:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280858368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829210.982434:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280858496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829210.982445:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829210.982450:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829210/real 1787829210] req@ffff8ab3f043bc00 x1874671280858496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829210.982458:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280858496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829210.982464:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829210.982465:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829210.982467:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280858496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829210.982471:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280858624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829210.982481:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829210.982486:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829210/real 1787829210] req@ffff8ab3f043ad00 x1874671280858624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829210.982494:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280858624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829210.982503:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829210.982507:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829210.982510:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280858624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829210.982516:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280858752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829210.982534:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829210.982546:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829210/real 1787829210] req@ffff8ab3f04383c0 x1874671280858752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829210.982557:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280858752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829210.982565:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829210.982567:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829210.982569:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280858752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829212.006128:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829213.030050:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829214.054115:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829215.078110:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829216.102056:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829216.102069:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829216.102074:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829216.102086:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3765 00000100:00080000:3.0:1787829216.102100:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829216.102109:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829216.102133:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829216.102175:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280858880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829216.102200:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829216.102204:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829216.102206:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829216.102210:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3765 00000100:00080000:3.0:1787829216.102215:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829216.102219:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829216.102232:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829216.102240:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280858880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829216.102252:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280859008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829216.102262:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829216.102265:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829216.102267:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829216.102271:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3765 00000100:00080000:3.0:1787829216.102274:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829216.102278:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829216.102283:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829216.102293:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280859136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829216.102302:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829216.102305:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829216.102307:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829216.102311:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3765 00000100:00080000:3.0:1787829216.102314:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829216.102319:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829216.102324:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829216.102336:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280859264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829216.102346:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829216.102350:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829216.102353:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829216.102358:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3765 00000100:00080000:3.0:1787829216.102361:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829216.102365:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829216.102369:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829216.102371:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280859008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787829216.102383:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280859392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829216.102414:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280859136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829216.102445:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280859264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829216.102470:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280859392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829217.126048:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829218.150051:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829219.174057:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787829219.174150:0:10942:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:3.0:1787829219.174192:0:10942:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829219.174208:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787829219.174240:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787829219.174249:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787829219.174257:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:2.0:1787829219.174263:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829216/real 1787829219] req@ffff8ab3d0a40000 x1874671280859008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787829219.174265:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:2.0:1787829219.174290:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280859008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829219.174304:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829219.174309:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829219.174313:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280859008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829219.174321:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829216/real 1787829219] req@ffff8ab3d0a43840 x1874671280859136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829219.174335:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280859136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829219.174344:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829219.174347:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829219.174352:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280859136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829219.174358:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829216/real 1787829219] req@ffff8ab3d0a412c0 x1874671280859264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829219.174370:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280859264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829219.174380:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829219.174383:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829219.174386:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280859264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829219.174393:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829216/real 1787829219] req@ffff8ab3d0a421c0 x1874671280858880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829219.174406:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280858880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829219.174415:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829219.174418:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829219.174422:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280858880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829219.174449:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829216/real 1787829219] req@ffff8ab3d0a43480 x1874671280859392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829219.174463:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280859392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829219.174474:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829219.174478:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829219.174501:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280859392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829220.198102:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829221.221998:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829221.222008:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829221.222012:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829221.222022:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3770 00000100:00080000:0.0:1787829221.222033:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829221.222039:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829221.222056:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829221.222092:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280859520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829221.222118:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829221.222121:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829221.222122:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829221.222125:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3770 00000100:00080000:0.0:1787829221.222128:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829221.222131:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829221.222141:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829221.222152:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280859648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829221.222158:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280859520:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829221.222171:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829221.222173:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829221.222176:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829221.222179:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3770 00000100:00080000:0.0:1787829221.222181:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829221.222184:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829221.222188:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829221.222198:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280859776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829221.222209:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829221.222213:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829221.222215:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829221.222219:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3770 00000100:00080000:0.0:1787829221.222232:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829221.222237:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829221.222242:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787829221.222246:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829221.222259:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280859904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829221.222270:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829221.222274:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829221.222276:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829221.222281:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3770 00000100:00080000:0.0:1787829221.222285:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829221.222289:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829221.222295:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829221.222310:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280860032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829221.222321:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829221/real 1787829221] req@ffff8ab3f075f840 x1874671280859520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829221.222346:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280859520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829221.222359:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829221.222364:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829221.222369:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280859520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829221.222399:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280859648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829221.222425:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829221.222441:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829221/real 1787829221] req@ffff8ab3f075e940 x1874671280859648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829221.222456:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280859648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829221.222472:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829221.222476:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829221.222479:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280859648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829221.222486:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280859776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829221.222507:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829221.222519:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829221/real 1787829221] req@ffff8ab3f075ed00 x1874671280859776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829221.222531:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280859776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829221.222539:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829221.222541:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829221.222544:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280859776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829221.222550:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280859904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829221.222569:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829221.222581:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829221/real 1787829221] req@ffff8ab3f075da40 x1874671280859904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829221.222592:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280859904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829221.222602:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829221.222604:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829221.222614:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280859904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829221.222621:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280860032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829221.222640:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829221.222649:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829221/real 1787829221] req@ffff8ab3f075f480 x1874671280860032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829221.222657:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280860032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829221.222666:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829221.222669:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829221.222673:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280860032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829222.246100:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829223.270108:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829224.294025:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829225.318009:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829226.342030:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829226.342039:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829226.342057:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829226.342067:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3775 00000100:00080000:0.0:1787829226.342077:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829226.342084:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829226.342084:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829226.342121:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280860160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829226.342142:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829226.342145:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829226.342146:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829226.342149:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3775 00000100:00080000:0.0:1787829226.342152:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829226.342155:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829226.342163:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829226.342173:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280860288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829226.342178:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280860160:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829226.342185:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829226.342190:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829226.342192:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829226.342196:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3775 00000100:00080000:0.0:1787829226.342198:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829226.342201:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829226.342204:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829226.342213:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280860416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829226.342219:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829226.342222:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829226.342223:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829226.342226:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3775 00000100:00080000:0.0:1787829226.342242:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829226.342245:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829226.342249:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787829226.342255:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829226.342261:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280860544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829226.342270:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829226.342272:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829226.342274:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829226.342277:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3775 00000100:00080000:0.0:1787829226.342279:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829226.342281:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829226.342284:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829226.342292:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280860672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829226.342310:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829226/real 1787829226] req@ffff8ab3f075e580 x1874671280860160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829226.342332:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280860160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829226.342344:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829226.342348:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829226.342352:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280860160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829226.342375:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280860288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829226.342411:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829226.342425:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829226/real 1787829226] req@ffff8ab3f075f480 x1874671280860288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829226.342437:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280860288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829226.342446:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829226.342449:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829226.342453:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280860288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829226.342459:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280860416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829226.342476:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829226.342486:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829226/real 1787829226] req@ffff8ab3f075da40 x1874671280860416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829226.342498:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280860416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829226.342506:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829226.342509:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829226.342513:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280860416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829226.342519:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280860544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829226.342535:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829226.342544:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829226/real 1787829226] req@ffff8ab3f075ed00 x1874671280860544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829226.342562:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280860544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829226.342571:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829226.342574:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829226.342577:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280860544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829226.342583:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280860672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829226.342599:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829226.342608:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829226/real 1787829226] req@ffff8ab3f075e940 x1874671280860672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829226.342620:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280860672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829226.342630:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829226.342634:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829226.342637:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280860672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829227.366084:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829228.390059:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829229.414072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829230.438034:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829231.462063:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829231.462071:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829231.462074:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829231.462083:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3781 00000100:00080000:3.0:1787829231.462093:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829231.462099:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829231.462109:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829231.462134:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280860800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829231.462155:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829231.462158:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829231.462159:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829231.462162:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3781 00000100:00080000:3.0:1787829231.462166:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829231.462169:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829231.462177:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829231.462181:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280860800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829231.462189:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280860928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829231.462199:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829231.462204:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829231.462205:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829231.462208:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3781 00000100:00080000:3.0:1787829231.462211:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829231.462213:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829231.462216:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829231.462224:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280861056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829231.462231:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829231.462233:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829231.462234:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829231.462240:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829231.462240:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3781 00000100:00080000:3.0:1787829231.462246:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829231.462250:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829231.462254:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829231.462263:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280861184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829231.462269:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829231.462272:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829231.462273:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829231.462276:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3781 00000100:00080000:3.0:1787829231.462279:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829231.462282:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829231.462286:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829231.462287:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829231/real 1787829231] req@ffff8ab3d0a41e00 x1874671280860800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829231.462310:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280860800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829231.462312:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280861312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829231.462331:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829231.462336:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829231.462341:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280860800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829231.462360:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280860928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829231.462374:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829231.462383:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829231/real 1787829231] req@ffff8ab3d0a430c0 x1874671280860928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829231.462391:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280860928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829231.462397:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829231.462399:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829231.462401:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280860928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829231.462405:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280861056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829231.462414:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829231.462419:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829231/real 1787829231] req@ffff8ab3d0a43c00 x1874671280861056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829231.462427:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280861056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829231.462436:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829231.462438:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829231.462441:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280861056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829231.462444:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280861184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829231.462453:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829231.462459:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829231/real 1787829231] req@ffff8ab3d0a43480 x1874671280861184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829231.462466:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280861184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829231.462472:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829231.462474:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829231.462476:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280861184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829231.462479:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280861312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829231.462488:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829231.462494:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829231/real 1787829231] req@ffff8ab3d0a421c0 x1874671280861312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829231.462501:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280861312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829231.462507:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829231.462508:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829231.462511:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280861312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829232.486061:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829233.510100:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829234.534044:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829235.558110:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829236.582029:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829236.582045:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829236.582050:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829236.582064:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3786 00000100:00080000:0.0:1787829236.582078:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829236.582087:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829236.582092:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829236.582141:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280861440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829236.582168:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829236.582171:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829236.582172:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829236.582175:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3786 00000100:00080000:0.0:1787829236.582179:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829236.582182:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829236.582192:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829236.582206:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280861568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829236.582211:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280861440:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829236.582222:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829236.582230:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829236.582232:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829236.582236:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3786 00000100:00080000:0.0:1787829236.582237:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829236.582240:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829236.582246:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829236.582256:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280861696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829236.582264:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829236.582267:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829236.582270:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829236.582273:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3786 00000100:00080000:0.0:1787829236.582285:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829236.582288:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787829236.582297:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787829236.582299:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829236.582317:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280861824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829236.582328:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829236.582332:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829236.582334:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829236.582341:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3786 00000100:00080000:0.0:1787829236.582346:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829236.582351:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829236.582358:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829236.582365:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829236/real 1787829236] req@ffff8ab3f075f840 x1874671280861440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829236.582376:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280861952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829236.582407:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280861440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829236.582421:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829236.582426:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829236.582431:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280861440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829236.582455:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280861568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829236.582479:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829236.582491:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829236/real 1787829236] req@ffff8ab3f075cb40 x1874671280861568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829236.582505:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280861568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829236.582514:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829236.582518:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829236.582522:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280861568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829236.582529:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280861696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829236.582546:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829236.582555:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829236/real 1787829236] req@ffff8ab3f075f0c0 x1874671280861696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829236.582568:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280861696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829236.582576:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829236.582580:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829236.582583:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280861696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829236.582590:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280861824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829236.582606:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829236.582616:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829236/real 1787829236] req@ffff8ab3f075de00 x1874671280861824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829236.582627:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280861824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829236.582638:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829236.582641:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829236.582645:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280861824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829236.582651:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280861952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829236.582666:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829236.582676:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829236/real 1787829236] req@ffff8ab3f075cf00 x1874671280861952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829236.582688:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280861952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829236.582698:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829236.582701:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829236.582705:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280861952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829237.606109:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829238.630050:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829239.654095:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829240.678038:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829241.702060:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829241.702071:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829241.702076:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829241.702088:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3791 00000100:00080000:2.0:1787829241.702099:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829241.702107:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829241.702139:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829241.702150:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280862080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829241.702185:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829241.702189:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829241.702191:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829241.702195:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3791 00000100:00080000:2.0:1787829241.702199:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829241.702203:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829241.702216:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829241.702235:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280862080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829241.702246:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280862208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829241.702260:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829241.702264:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829241.702266:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829241.702271:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3791 00000100:00080000:2.0:1787829241.702274:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829241.702278:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829241.702283:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829241.702296:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280862336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829241.702306:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829241.702314:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829241.702315:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829241.702321:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829241.702326:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3791 00000100:00080000:2.0:1787829241.702332:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829241.702337:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829241.702343:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829241.702355:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280862464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829241.702366:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829241.702370:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829241.702372:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829241.702379:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829241/real 1787829241] req@ffff8ab3f04392c0 x1874671280862080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829241.702382:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3791 00000100:00080000:2.0:1787829241.702391:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829241.702401:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829241.702407:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829241.702414:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280862080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829241.702422:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280862592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829241.702437:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829241.702445:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829241.702451:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280862080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829241.702479:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280862208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829241.702501:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829241.702530:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829241/real 1787829241] req@ffff8ab3f043a940 x1874671280862208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829241.702540:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280862208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829241.702547:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829241.702549:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829241.702552:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280862208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829241.702557:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280862336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829241.702570:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829241.702577:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829241/real 1787829241] req@ffff8ab3f043a1c0 x1874671280862336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829241.702586:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280862336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829241.702592:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829241.702594:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829241.702596:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280862336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829241.702600:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280862464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829241.702612:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829241.702619:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829241/real 1787829241] req@ffff8ab3f0439e00 x1874671280862464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829241.702628:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280862464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829241.702635:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829241.702637:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829241.702639:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280862464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829241.702643:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280862592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829241.702654:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829241.702661:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829241/real 1787829241] req@ffff8ab3f043b480 x1874671280862592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829241.702670:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280862592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829241.702677:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829241.702679:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829241.702681:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280862592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829242.726062:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829243.750108:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829244.774087:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829245.798078:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829246.822035:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829246.822048:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829246.822064:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829246.822080:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3796 00000100:00080000:3.0:1787829246.822097:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829246.822107:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829246.822153:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280862720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787829246.822165:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829246.822183:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829246.822189:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829246.822191:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829246.822196:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3796 00000100:00080000:3.0:1787829246.822199:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829246.822203:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829246.822214:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829246.822229:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280862848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829246.822254:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829246.822266:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829246.822270:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280862720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829246.822272:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829246.822293:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3796 00000100:00080000:3.0:1787829246.822298:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829246.822303:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829246.822308:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829246.822322:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280862976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829246.822334:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829246.822338:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829246.822340:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829246.822345:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3796 00000100:00080000:3.0:1787829246.822349:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829246.822353:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829246.822366:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829246.822380:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280863104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829246.822390:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829246.822395:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829246.822398:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829246.822413:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829246.822421:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3796 00000100:00080000:3.0:1787829246.822445:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829246.822451:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829246.822460:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829246.822480:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280863232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829246.822548:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829246/real 1787829246] req@ffff8ab3d0a412c0 x1874671280862720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829246.822577:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280862720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829246.822605:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829246.822611:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829246.822617:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280862720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829246.822679:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280862848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829246.822727:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829246.822776:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829246/real 1787829246] req@ffff8ab3d0a43840 x1874671280862848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829246.822794:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280862848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829246.822806:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829246.822811:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829246.822815:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280862848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829246.822824:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280862976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829246.822854:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829246.822867:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829246/real 1787829246] req@ffff8ab3d0a40000 x1874671280862976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829246.822881:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280862976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829246.822955:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829246.822960:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829246.822966:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280862976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829246.822975:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280863104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829246.823007:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829246.823023:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829246/real 1787829246] req@ffff8ab3d0a421c0 x1874671280863104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829246.823038:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280863104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829246.823050:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829246.823054:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829246.823058:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280863104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829246.823066:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280863232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829246.823085:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829246.823096:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829246/real 1787829246] req@ffff8ab3d0a43480 x1874671280863232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829246.823110:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280863232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829246.823122:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829246.823125:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829246.823129:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280863232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829247.846245:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829248.870091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829249.894103:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829250.918089:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829251.941980:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829251.941989:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829251.941993:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829251.942002:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3801 00010000:00010000:3.0:1787829251.942010:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829251.942012:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829251.942018:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829251.942050:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280863360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829251.942071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829251.942073:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829251.942075:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829251.942077:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3801 00000100:00080000:0.0:1787829251.942080:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829251.942082:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829251.942090:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829251.942100:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280863488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829251.942108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829251.942112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829251.942115:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829251.942115:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280863360:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829251.942123:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3801 00000100:00080000:0.0:1787829251.942126:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829251.942129:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829251.942132:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829251.942141:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280863616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829251.942152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829251.942155:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829251.942157:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829251.942161:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3801 00000100:00080000:0.0:1787829251.942174:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829251.942181:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829251.942183:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787829251.942190:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829251.942205:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280863744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829251.942215:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829251.942217:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829251.942219:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829251.942222:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3801 00000100:00080000:0.0:1787829251.942225:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829251.942228:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829251.942233:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829251.942241:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280863872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829251.942251:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829251/real 1787829251] req@ffff8ab3f075c000 x1874671280863360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829251.942269:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280863360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829251.942277:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829251.942280:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829251.942283:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280863360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829251.942302:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280863488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829251.942320:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829251.942329:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829251/real 1787829251] req@ffff8ab3f075cf00 x1874671280863488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829251.942338:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280863488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829251.942343:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829251.942345:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829251.942347:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280863488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829251.942351:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280863616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829251.942361:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829251.942366:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829251/real 1787829251] req@ffff8ab3f075de00 x1874671280863616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829251.942374:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280863616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829251.942384:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829251.942386:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829251.942388:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280863616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829251.942391:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280863744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829251.942401:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829251.942406:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829251/real 1787829251] req@ffff8ab3f075f0c0 x1874671280863744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829251.942413:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280863744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829251.942419:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829251.942421:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829251.942423:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280863744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829251.942427:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280863872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829251.942442:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829251.942449:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829251/real 1787829251] req@ffff8ab3f075cb40 x1874671280863872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829251.942457:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280863872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829251.942463:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829251.942465:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829251.942467:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280863872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829252.966108:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829253.990123:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829255.014103:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829256.038048:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829257.062002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829257.062014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829257.062019:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829257.062031:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3806 00000100:00080000:0.0:1787829257.062043:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829257.062052:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829257.062096:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280864000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787829257.062105:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829257.062127:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829257.062131:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829257.062133:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829257.062136:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3806 00000100:00080000:0.0:1787829257.062139:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829257.062141:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829257.062151:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829257.062160:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280864128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829257.062168:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829257.062171:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829257.062172:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829257.062178:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3806 00000100:00080000:0.0:1787829257.062187:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829257.062190:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829257.062194:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829257.062202:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280864000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829257.062204:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280864256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829257.062227:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829257.062230:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829257.062233:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829257.062239:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3806 00000100:00080000:0.0:1787829257.062242:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829257.062245:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829257.062249:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829257.062258:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280864384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829257.062265:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829257.062267:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829257.062268:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829257.062271:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3806 00000100:00080000:0.0:1787829257.062276:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829257.062279:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829257.062283:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787829257.062287:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829257.062293:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280864512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829257.062342:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829257/real 1787829257] req@ffff8ab3f075f840 x1874671280864000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829257.062358:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280864000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829257.062368:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829257.062373:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829257.062377:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280864000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829257.062385:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280864128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829257.062402:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829257.062410:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829257/real 1787829257] req@ffff8ab3f075e940 x1874671280864128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829257.062418:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280864128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829257.062423:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829257.062425:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829257.062427:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280864128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829257.062454:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280864256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829257.062466:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829257.062474:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829257/real 1787829257] req@ffff8ab3f075ed00 x1874671280864256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829257.062481:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280864256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829257.062487:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829257.062489:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829257.062491:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280864256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829257.062495:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280864384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829257.062506:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829257.062511:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829257/real 1787829257] req@ffff8ab3f075da40 x1874671280864384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829257.062518:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280864384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829257.062524:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829257.062526:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829257.062528:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280864384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829257.062532:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280864512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829257.062542:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829257.062554:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829257/real 1787829257] req@ffff8ab3f075f480 x1874671280864512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829257.062561:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280864512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829257.062568:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829257.062570:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829257.062572:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280864512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829258.086045:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829259.110092:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829260.134180:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829261.158140:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829262.182000:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829262.182010:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829262.182014:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829262.182024:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3811 00000100:00080000:0.0:1787829262.182035:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829262.182042:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829262.182075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829262.182083:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280864640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829262.182112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829262.182117:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829262.182120:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829262.182126:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3811 00000100:00080000:0.0:1787829262.182132:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829262.182139:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829262.182152:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829262.182159:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280864640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829262.182176:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280864768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829262.182190:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829262.182193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829262.182195:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829262.182199:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3811 00000100:00080000:0.0:1787829262.182222:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829262.182226:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829262.182231:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829262.182233:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829262.182248:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280864896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829262.182258:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829262.182261:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829262.182263:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829262.182267:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3811 00000100:00080000:0.0:1787829262.182271:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829262.182276:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829262.182281:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829262.182284:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829262/real 1787829262] req@ffff8ab3f075e580 x1874671280864640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829262.182294:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280865024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829262.182305:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829262.182306:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280864640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829262.182313:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829262.182317:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829262.182327:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3811 00000100:00080000:1.0:1787829262.182329:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829262.182331:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829262.182338:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829262.182341:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787829262.182347:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787829262.182352:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280865152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829262.182357:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280864640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829262.182389:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280864768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829262.182414:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829262.182429:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829262/real 1787829262] req@ffff8ab3f075f480 x1874671280864768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829262.182443:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280864768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829262.182451:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829262.182454:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829262.182457:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280864768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829262.182462:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280864896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829262.182484:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829262.182493:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829262/real 1787829262] req@ffff8ab3f075da40 x1874671280864896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829262.182502:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280864896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829262.182509:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829262.182511:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829262.182514:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280864896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829262.182519:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280865024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829262.182531:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829262.182538:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829262/real 1787829262] req@ffff8ab3f075ed00 x1874671280865024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829262.182547:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280865024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829262.182555:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829262.182557:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829262.182559:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280865024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829262.182563:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280865152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829262.182574:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829262.182581:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829262/real 1787829262] req@ffff8ab3f075e940 x1874671280865152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829262.182591:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280865152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829262.182599:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829262.182601:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829262.182603:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280865152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829263.206050:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829264.230149:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829265.254027:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829266.278017:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829267.302117:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829267.302139:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829267.302148:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1787829267.302167:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829267.302169:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3816 00000100:00080000:3.0:1787829267.302193:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829267.302209:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829267.302284:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280865280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829267.302334:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829267.302341:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829267.302345:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829267.302354:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3816 00000100:00080000:3.0:1787829267.302363:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829267.302376:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829267.302396:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829267.302403:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280865280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829267.302430:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280865408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829267.302458:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829267.302465:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829267.302468:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829267.302475:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3816 00000100:00080000:3.0:1787829267.302485:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829267.302492:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829267.302501:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829267.302503:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829267.302530:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280865536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829267.302546:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829267.302551:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829267.302553:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829267.302559:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3816 00000100:00080000:3.0:1787829267.302571:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829267.302577:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829267.302585:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829267.302605:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280865664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829267.302619:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829267.302622:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829267/real 1787829267] req@ffff8ab3d0a43c00 x1874671280865280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829267.302629:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829267.302636:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829267.302653:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3816 00000100:00080000:3.0:1787829267.302660:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829267.302665:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829267.302673:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829267.302679:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280865280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829267.302696:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280865792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829267.302703:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829267.302720:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829267.302731:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280865280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829267.302802:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280865408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829267.302855:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829267.302883:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829267/real 1787829267] req@ffff8ab3d0a430c0 x1874671280865408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829267.302936:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280865408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829267.302958:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829267.302963:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829267.302968:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280865408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829267.302977:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280865536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829267.303007:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829267.303026:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829267/real 1787829267] req@ffff8ab3d0a41e00 x1874671280865536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829267.303049:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280865536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829267.303066:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829267.303073:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829267.303080:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280865536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829267.303093:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280865664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829267.303132:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829267.303148:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829267/real 1787829267] req@ffff8ab3d0a43480 x1874671280865664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829267.303189:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280865664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829267.303209:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829267.303216:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829267.303225:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280865664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829267.303237:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280865792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829267.303274:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829267.303290:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829267/real 1787829267] req@ffff8ab3d0a421c0 x1874671280865792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829267.303305:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280865792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829267.303318:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829267.303322:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829267.303327:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280865792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829268.326111:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829269.350126:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829270.374134:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829271.398026:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829272.422083:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829272.422125:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829272.422135:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829272.422139:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829272.422156:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3822 00000100:00080000:0.0:1787829272.422165:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829272.422172:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829272.422207:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280865920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829272.422246:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829272.422251:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829272.422252:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829272.422255:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3822 00000100:00080000:0.0:1787829272.422258:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829272.422261:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829272.422270:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829272.422284:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280866048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829272.422288:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280865920:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829272.422306:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829272.422310:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829272.422313:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829272.422319:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3822 00000100:00080000:0.0:1787829272.422323:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829272.422327:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829272.422331:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829272.422343:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280866176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829272.422357:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829272.422360:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829272.422362:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829272.422367:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3822 00000800:00000100:1.0:1787829272.422370:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829272.422371:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829272.422374:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829272.422378:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829272.422386:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280866304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829272.422395:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829272.422397:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829272.422399:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829272.422402:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3822 00000100:00080000:0.0:1787829272.422403:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829272.422406:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829272.422409:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829272.422416:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280866432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829272.422432:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829272/real 1787829272] req@ffff8ab3f075f840 x1874671280865920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829272.422451:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280865920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829272.422463:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829272.422467:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829272.422471:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280865920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829272.422512:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280866048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829272.422538:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829272.422577:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829272/real 1787829272] req@ffff8ab3f075cb40 x1874671280866048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829272.422593:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280866048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829272.422604:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829272.422607:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829272.422612:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280866048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829272.422619:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280866176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829272.422639:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829272.422648:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829272/real 1787829272] req@ffff8ab3f075f0c0 x1874671280866176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829272.422657:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280866176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829272.422664:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829272.422666:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829272.422668:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280866176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829272.422672:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280866304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829272.422685:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829272.422691:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829272/real 1787829272] req@ffff8ab3f075de00 x1874671280866304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829272.422699:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280866304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829272.422706:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829272.422708:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829272.422711:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280866304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829272.422715:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280866432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829272.422744:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829272.422753:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829272/real 1787829272] req@ffff8ab3f075cf00 x1874671280866432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829272.422762:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280866432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829272.422769:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829272.422772:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829272.422774:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280866432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829273.446012:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829274.470065:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829275.494018:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829276.518066:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829277.542031:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829277.542041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1787829277.542046:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829277.542047:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829277.542060:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3827 00000100:00080000:0.0:1787829277.542072:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829277.542078:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829277.542114:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280866560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829277.542137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829277.542141:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829277.542143:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829277.542146:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3827 00000100:00080000:0.0:1787829277.542149:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829277.542154:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829277.542164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829277.542178:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280866688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829277.542192:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280866560:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829277.542201:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829277.542205:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829277.542206:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829277.542210:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3827 00000100:00080000:0.0:1787829277.542214:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829277.542217:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829277.542221:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829277.542232:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280866816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829277.542240:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829277.542243:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829277.542244:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829277.542248:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3827 00000100:00080000:0.0:1787829277.542252:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829277.542272:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829277.542279:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829277.542287:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829277.542305:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280866944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829277.542333:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829277/real 1787829277] req@ffff8ab3f075c000 x1874671280866560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829277.542337:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829277.542348:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829277.542353:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829277.542363:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3827 00000100:00080000:0.0:1787829277.542369:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829277.542376:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829277.542378:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280866560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829277.542382:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787829277.542400:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829277.542407:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280867072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829277.542407:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829277.542420:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280866560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829277.542457:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280866688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829277.542484:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829277.542497:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829277/real 1787829277] req@ffff8ab3f075cf00 x1874671280866688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829277.542508:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280866688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829277.542515:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829277.542518:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829277.542520:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280866688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829277.542526:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280866816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829277.542550:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829277.542557:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829277/real 1787829277] req@ffff8ab3f075de00 x1874671280866816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829277.542566:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280866816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829277.542572:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829277.542574:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829277.542577:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280866816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829277.542581:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280866944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829277.542593:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829277.542610:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829277/real 1787829277] req@ffff8ab3f075f0c0 x1874671280866944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829277.542619:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280866944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829277.542626:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829277.542628:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829277.542631:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280866944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829277.542635:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280867072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829277.542645:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829277.542652:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829277/real 1787829277] req@ffff8ab3f075cb40 x1874671280867072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829277.542661:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280867072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829277.542668:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829277.542670:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829277.542672:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280867072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829278.566118:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829279.590071:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829280.614110:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829281.638014:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829282.662011:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829282.662020:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829282.662024:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829282.662034:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3832 00000100:00080000:0.0:1787829282.662044:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829282.662050:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829282.662084:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280867200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829282.662119:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829282.662124:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829282.662126:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829282.662132:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3832 00010000:00010000:3.0:1787829282.662137:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829282.662138:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829282.662148:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829282.662152:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280867200:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787829282.662163:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829282.662189:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280867328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829282.662202:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829282.662206:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829282.662208:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829282.662225:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3832 00000100:00080000:0.0:1787829282.662230:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829282.662235:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829282.662240:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829282.662252:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280867456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829282.662259:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829282.662262:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829282.662264:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829282.662267:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3832 00000100:00080000:0.0:1787829282.662270:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829282.662274:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280867328:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787829282.662276:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829282.662283:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829282.662295:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280867584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829282.662306:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829282.662311:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829282.662313:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280867456:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787829282.662315:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829282.662324:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3832 00000100:00080000:0.0:1787829282.662330:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829282.662336:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829282.662341:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280867584:192.168.203.124@tcp:8:kworker.0 00000008:00100000:0.0:1787829282.662343:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829282.662363:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280867712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829282.662380:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280867712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829283.686085:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829284.710197:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829285.734172:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829285.734322:0:10944:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787829285.734373:0:10944:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829285.734394:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829285.734440:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829285.734453:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829285.734467:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829285.734479:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787829285.734491:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829282/real 1787829285] req@ffff8ab3f075da40 x1874671280867584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829285.734527:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280867584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829285.734545:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829285.734552:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829285.734558:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280867584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829285.734567:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829282/real 1787829285] req@ffff8ab3f075e940 x1874671280867328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829285.734581:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280867328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829285.734592:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829285.734596:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829285.734600:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280867328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829285.734607:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829282/real 1787829285] req@ffff8ab3f075ed00 x1874671280867456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829285.734618:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280867456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829285.734646:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829285.734651:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829285.734655:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280867456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829285.734662:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829282/real 1787829285] req@ffff8ab3f075f840 x1874671280867200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829285.734674:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280867200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829285.734682:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829285.734685:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829285.734688:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280867200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829285.734716:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829282/real 1787829285] req@ffff8ab3f075f480 x1874671280867712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829285.734728:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280867712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829285.734737:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829285.734741:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829285.734744:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280867712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829286.758131:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829287.782065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829287.782081:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1787829287.782085:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829287.782086:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829287.782101:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3837 00000100:00080000:2.0:1787829287.782114:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829287.782121:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829287.782170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280867840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829287.782210:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829287.782214:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829287.782216:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829287.782232:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3837 00000100:00080000:2.0:1787829287.782236:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829287.782242:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829287.782254:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829287.782275:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280867968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829287.782280:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280867840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829287.782308:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829287.782313:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829287.782316:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829287.782324:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3837 00000100:00080000:2.0:1787829287.782348:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829287.782357:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829287.782359:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829287.782372:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829287.782395:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280868096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829287.782408:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829287.782410:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829287/real 1787829287] req@ffff8ab3f043b840 x1874671280867840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829287.782421:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829287.782427:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829287.782438:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3837 00000100:00080000:2.0:1787829287.782447:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829287.782457:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829287.782465:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280867840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829287.782466:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787829287.782490:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829287.782494:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280868224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829287.782505:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787829287.782514:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829287.782516:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280867840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829287.782527:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829287.782531:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829287.782539:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3837 00000100:00080000:2.0:1787829287.782545:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829287.782552:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280867968:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787829287.782554:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829287.782567:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829287.782589:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280868352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829287.782605:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829287.782658:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829287/real 1787829287] req@ffff8ab3f043b480 x1874671280867968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829287.782677:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280867968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829287.782687:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829287.782691:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829287.782695:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280867968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829287.782703:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280868096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829287.782726:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829287.782739:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829287/real 1787829287] req@ffff8ab3f0439e00 x1874671280868096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829287.782752:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280868096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829287.782760:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829287.782764:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829287.782768:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280868096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829287.782784:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280868224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829287.782813:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829287.782837:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829287/real 1787829287] req@ffff8ab3f043a1c0 x1874671280868224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829287.782851:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280868224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829287.782862:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829287.782866:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829287.782870:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280868224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829287.782921:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280868352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829287.782944:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829287.782956:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829287/real 1787829287] req@ffff8ab3f043a940 x1874671280868352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829287.782967:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280868352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829287.782977:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829287.782980:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829287.782984:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280868352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829288.806092:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829289.830036:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829290.854200:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829291.878102:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829292.901990:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829292.901998:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829292.902002:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829292.902012:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3842 00000100:00080000:3.0:1787829292.902021:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829292.902027:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829292.902029:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829292.902058:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280868480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829292.902078:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829292.902081:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829292.902082:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829292.902085:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3842 00000100:00080000:3.0:1787829292.902087:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829292.902090:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829292.902103:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829292.902112:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280868480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829292.902116:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280868608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829292.902126:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829292.902129:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829292.902130:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829292.902133:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3842 00000100:00080000:3.0:1787829292.902135:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829292.902138:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829292.902142:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829292.902151:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280868736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829292.902157:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829292.902160:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829292.902161:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829292.902163:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3842 00000100:00080000:3.0:1787829292.902168:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829292.902171:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829292.902176:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829292.902180:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829292.902189:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280868864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829292.902197:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829292.902199:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829292.902200:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829292.902203:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3842 00000100:00080000:3.0:1787829292.902205:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829292.902213:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829292.902217:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829292.902225:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829292/real 1787829292] req@ffff8ab3d0a40000 x1874671280868480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829292.902228:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280868992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829292.902265:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280868480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829292.902274:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829292.902278:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829292.902281:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280868480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829292.902300:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280868608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829292.902315:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829292.902324:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829292/real 1787829292] req@ffff8ab3d0a43840 x1874671280868608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829292.902332:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280868608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829292.902338:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829292.902340:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829292.902343:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280868608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829292.902346:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280868736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829292.902356:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829292.902362:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829292/real 1787829292] req@ffff8ab3d0a412c0 x1874671280868736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829292.902369:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280868736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829292.902379:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829292.902382:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829292.902386:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280868736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829292.902392:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280868864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829292.902411:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829292.902423:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829292/real 1787829292] req@ffff8ab3d0a421c0 x1874671280868864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829292.902436:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280868864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829292.902444:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829292.902446:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829292.902448:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280868864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829292.902453:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280868992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829292.902467:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829292.902475:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829292/real 1787829292] req@ffff8ab3d0a43480 x1874671280868992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829292.902482:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280868992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829292.902488:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829292.902490:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829292.902492:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280868992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829293.926119:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829294.950053:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829295.974078:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829296.998102:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829298.022004:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829298.022013:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829298.022018:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829298.022027:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3847 00000100:00080000:0.0:1787829298.022036:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829298.022043:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829298.022049:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829298.022089:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280869120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829298.022113:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829298.022115:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829298.022117:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829298.022120:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3847 00000100:00080000:0.0:1787829298.022122:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829298.022125:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829298.022133:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829298.022144:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280869248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829298.022153:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280869120:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829298.022155:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829298.022162:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829298.022163:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829298.022167:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3847 00000100:00080000:0.0:1787829298.022168:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829298.022171:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829298.022174:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829298.022183:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280869376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829298.022188:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829298.022191:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829298.022192:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829298.022195:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3847 00000100:00080000:0.0:1787829298.022201:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829298.022204:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829298.022207:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829298.022212:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829298.022218:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280869504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829298.022226:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829298.022229:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829298.022230:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829298.022233:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3847 00000100:00080000:0.0:1787829298.022235:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829298.022237:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829298.022240:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829298.022249:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280869632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829298.022255:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829298/real 1787829298] req@ffff8ab3f075e580 x1874671280869120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829298.022269:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280869120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829298.022278:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829298.022282:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829298.022284:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280869120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829298.022303:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280869248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829298.022335:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829298.022346:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829298/real 1787829298] req@ffff8ab3f075f480 x1874671280869248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829298.022355:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280869248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829298.022362:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829298.022364:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829298.022367:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280869248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829298.022371:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280869376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829298.022383:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829298.022390:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829298/real 1787829298] req@ffff8ab3f075f840 x1874671280869376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829298.022408:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280869376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829298.022414:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829298.022416:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829298.022418:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280869376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829298.022422:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280869504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829298.022433:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829298.022440:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829298/real 1787829298] req@ffff8ab3f075ed00 x1874671280869504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829298.022459:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280869504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829298.022470:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829298.022472:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829298.022475:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280869504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829298.022479:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280869632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829298.022492:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829298.022499:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829298/real 1787829298] req@ffff8ab3f075e940 x1874671280869632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829298.022507:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280869632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829298.022515:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829298.022517:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829298.022520:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280869632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829299.046048:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829300.070101:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829301.094079:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829302.118073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829303.142088:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829303.142090:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829303.142100:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829303.142104:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829303.142115:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3852 00000100:00080000:3.0:1787829303.142126:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829303.142133:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829303.142198:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280869760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829303.142225:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829303.142241:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829303.142242:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829303.142245:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3852 00000100:00080000:3.0:1787829303.142252:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829303.142255:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829303.142265:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829303.142275:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280869888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829303.142284:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829303.142288:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829303.142289:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829303.142294:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3852 00000100:00080000:3.0:1787829303.142296:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829303.142298:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829303.142302:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829303.142311:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280869760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829303.142312:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280870016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829303.142337:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829303.142339:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829303.142340:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829303.142344:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3852 00000100:00080000:3.0:1787829303.142347:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829303.142351:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829303.142356:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829303.142369:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280870144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829303.142377:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829303.142380:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829303.142381:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829303.142384:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3852 00000100:00080000:3.0:1787829303.142406:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829303.142409:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829303.142412:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787829303.142420:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829303.142425:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280870272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829303.142478:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829303/real 1787829303] req@ffff8ab3d0a41e00 x1874671280869760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829303.142496:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280869760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829303.142523:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829303.142527:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829303.142530:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280869760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829303.142585:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280869888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829303.142604:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829303.142623:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829303/real 1787829303] req@ffff8ab3d0a430c0 x1874671280869888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829303.142632:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280869888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829303.142639:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829303.142642:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829303.142644:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280869888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829303.142650:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280870016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829303.142661:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829303.142668:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829303/real 1787829303] req@ffff8ab3d0a43c00 x1874671280870016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829303.142677:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280870016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829303.142683:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829303.142685:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829303.142687:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280870016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829303.142691:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280870144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829303.142703:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829303.142709:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829303/real 1787829303] req@ffff8ab3d0a43480 x1874671280870144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829303.142717:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280870144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829303.142724:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829303.142726:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829303.142728:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280870144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829303.142733:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280870272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829303.142762:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829303.142770:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829303/real 1787829303] req@ffff8ab3d0a421c0 x1874671280870272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829303.142779:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280870272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829303.142786:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829303.142789:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829303.142791:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280870272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829304.166082:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829305.190110:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829306.214080:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829307.238074:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829308.262036:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829308.262045:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829308.262050:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829308.262058:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3857 00000100:00080000:2.0:1787829308.262070:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829308.262079:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829308.262090:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829308.262116:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280870400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829308.262143:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829308.262147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829308.262149:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829308.262155:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3857 00000100:00080000:2.0:1787829308.262159:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829308.262163:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829308.262171:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829308.262175:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280870400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829308.262186:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280870528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829308.262195:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829308.262199:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829308.262200:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829308.262203:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3857 00000100:00080000:2.0:1787829308.262220:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829308.262225:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829308.262231:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829308.262234:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829308.262253:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280870656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829308.262272:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829308.262273:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829308/real 1787829308] req@ffff8ab3f04392c0 x1874671280870400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829308.262277:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829308.262281:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829308.262289:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3857 00000100:00080000:2.0:1787829308.262294:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829308.262299:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829308.262305:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829308.262309:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280870400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829308.262318:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280870784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829308.262331:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829308.262332:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829308.262339:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829308.262340:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829308.262344:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787829308.262347:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3857 00000100:00080000:2.0:1787829308.262351:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829308.262355:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280870400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829308.262356:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829308.262362:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829308.262372:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280870912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829308.262387:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280870528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829308.262421:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829308.262431:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829308/real 1787829308] req@ffff8ab3f04383c0 x1874671280870528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829308.262439:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280870528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829308.262449:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829308.262452:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829308.262454:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280870528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829308.262458:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280870656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829308.262469:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829308.262493:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829308/real 1787829308] req@ffff8ab3f043ad00 x1874671280870656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829308.262501:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280870656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829308.262532:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829308.262534:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829308.262537:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280870656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829308.262541:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280870784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829308.262556:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829308.262563:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829308/real 1787829308] req@ffff8ab3f043bc00 x1874671280870784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829308.262570:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280870784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829308.262577:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829308.262596:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829308.262599:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280870784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829308.262604:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280870912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829308.262616:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829308.262623:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829308/real 1787829308] req@ffff8ab3f0439a40 x1874671280870912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829308.262631:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280870912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829308.262638:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829308.262640:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829308.262642:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280870912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829309.286109:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829310.310060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829311.334049:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829312.358074:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829313.382055:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829313.382063:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829313.382067:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829313.382077:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3863 00000100:00080000:0.0:1787829313.382086:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829313.382095:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829313.382121:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829313.382130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280871040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829313.382153:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829313.382158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829313.382160:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829313.382163:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3863 00000100:00080000:0.0:1787829313.382165:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829313.382168:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829313.382177:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829313.382187:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280871168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829313.382198:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829313.382202:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829313.382209:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829313.382210:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280871040:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829313.382223:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3863 00000100:00080000:0.0:1787829313.382227:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829313.382231:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829313.382235:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829313.382248:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280871296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829313.382256:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829313.382260:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829313.382262:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829313.382268:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3863 00000100:00080000:0.0:1787829313.382293:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829313.382299:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829313.382310:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787829313.382315:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829313.382334:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280871424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829313.382346:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829313.382350:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829313.382352:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829313.382357:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3863 00000100:00080000:0.0:1787829313.382362:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829313.382369:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829313.382375:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829313.382378:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829313/real 1787829313] req@ffff8ab3f075da40 x1874671280871040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829313.382393:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280871552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829313.382402:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280871040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829313.382425:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829313.382431:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829313.382448:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280871040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829313.382476:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280871168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829313.382497:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829313.382508:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829313/real 1787829313] req@ffff8ab3f075cb40 x1874671280871168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829313.382524:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280871168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829313.382531:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829313.382533:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829313.382536:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280871168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829313.382541:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280871296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829313.382554:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829313.382561:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829313/real 1787829313] req@ffff8ab3f075f0c0 x1874671280871296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829313.382569:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280871296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829313.382574:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829313.382577:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829313.382579:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280871296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829313.382583:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280871424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829313.382597:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829313.382607:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829313/real 1787829313] req@ffff8ab3f075de00 x1874671280871424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829313.382620:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280871424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829313.382632:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829313.382636:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829313.382640:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280871424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829313.382647:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280871552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829313.382665:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829313.382672:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829313/real 1787829313] req@ffff8ab3f075cf00 x1874671280871552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829313.382680:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280871552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829313.382688:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829313.382690:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829313.382692:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280871552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829314.406095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829315.430058:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829316.454037:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829317.478047:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829318.501987:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829318.501997:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829318.502001:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829318.502011:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3868 00000100:00080000:3.0:1787829318.502022:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829318.502038:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829318.502041:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829318.502075:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280871680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829318.502096:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829318.502100:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829318.502102:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829318.502106:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3868 00000100:00080000:3.0:1787829318.502110:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829318.502115:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829318.502124:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829318.502132:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280871680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829318.502135:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280871808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829318.502145:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829318.502148:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829318.502150:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829318.502153:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3868 00000100:00080000:3.0:1787829318.502156:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829318.502160:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829318.502164:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829318.502173:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280871936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829318.502180:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829318.502183:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829318.502186:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829318.502189:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829318.502189:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3868 00000100:00080000:3.0:1787829318.502192:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829318.502196:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829318.502200:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829318.502210:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280872064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829318.502218:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829318.502221:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829318.502223:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829318.502227:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3868 00000100:00080000:3.0:1787829318.502229:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829318.502233:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829318.502234:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829318/real 1787829318] req@ffff8ab3d0a412c0 x1874671280871680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829318.502240:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829318.502248:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280872192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829318.502253:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280871680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829318.502263:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829318.502267:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829318.502271:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280871680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829318.502295:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280871808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829318.502315:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829318.502325:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829318/real 1787829318] req@ffff8ab3d0a43840 x1874671280871808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829318.502333:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280871808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829318.502339:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829318.502341:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829318.502343:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280871808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829318.502347:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280871936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829318.502357:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829318.502362:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829318/real 1787829318] req@ffff8ab3d0a40000 x1874671280871936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829318.502369:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280871936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829318.502374:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829318.502376:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829318.502378:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280871936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829318.502381:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280872064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829318.502390:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829318.502395:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829318/real 1787829318] req@ffff8ab3d0a421c0 x1874671280872064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829318.502416:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280872064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829318.502423:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829318.502425:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829318.502427:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280872064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829318.502431:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280872192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829318.502443:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829318.502449:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829318/real 1787829318] req@ffff8ab3d0a43480 x1874671280872192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829318.502456:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280872192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829318.502462:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829318.502464:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829318.502466:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280872192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829319.526056:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829320.550011:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829321.574047:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829322.598037:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829323.622020:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829323.622031:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829323.622035:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829323.622047:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3873 00000100:00080000:0.0:1787829323.622061:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829323.622072:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829323.622081:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829323.622129:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280872320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829323.622157:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829323.622161:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829323.622163:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829323.622167:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3873 00000100:00080000:0.0:1787829323.622173:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829323.622179:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829323.622191:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829323.622200:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280872320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829323.622215:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280872448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829323.622227:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829323.622230:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829323.622232:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829323.622236:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3873 00000100:00080000:0.0:1787829323.622240:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829323.622243:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829323.622249:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829323.622259:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280872576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829323.622268:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829323.622271:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829323.622278:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829323.622280:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829323.622285:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3873 00000100:00080000:0.0:1787829323.622290:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829323.622293:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829323.622299:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829323.622311:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280872704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829323.622323:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829323.622330:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829323.622330:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829323/real 1787829323] req@ffff8ab3f075c000 x1874671280872320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829323.622336:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829323.622350:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3873 00000100:00080000:0.0:1787829323.622356:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829323.622362:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829323.622383:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829323.622383:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280872320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829323.622401:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829323.622402:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280872832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829323.622417:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829323.622433:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280872320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829323.622461:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280872448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829323.622480:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829323.622489:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829323/real 1787829323] req@ffff8ab3f075cf00 x1874671280872448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829323.622498:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280872448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829323.622503:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829323.622506:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829323.622508:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280872448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829323.622512:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280872576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829323.622521:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829323.622527:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829323/real 1787829323] req@ffff8ab3f075de00 x1874671280872576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829323.622534:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280872576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829323.622540:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829323.622541:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829323.622544:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280872576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829323.622548:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280872704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829323.622558:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829323.622564:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829323/real 1787829323] req@ffff8ab3f075f0c0 x1874671280872704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829323.622570:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280872704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829323.622577:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829323.622579:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829323.622580:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280872704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829323.622584:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280872832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829323.622594:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829323.622599:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829323/real 1787829323] req@ffff8ab3f075cb40 x1874671280872832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829323.622606:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280872832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829323.622612:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829323.622614:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829323.622616:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280872832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829324.646041:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829325.670060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829326.694095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829327.718045:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829328.742016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829328.742024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829328.742028:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829328.742037:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3878 00000100:00080000:2.0:1787829328.742047:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829328.742054:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829328.742068:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829328.742088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280872960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829328.742111:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829328.742114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829328.742116:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829328.742118:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3878 00000100:00080000:2.0:1787829328.742121:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829328.742124:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829328.742132:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829328.742144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280873088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829328.742145:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280872960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829328.742159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829328.742162:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829328.742163:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829328.742167:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3878 00000100:00080000:2.0:1787829328.742169:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829328.742171:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829328.742175:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829328.742184:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280873216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829328.742191:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829328.742194:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829328.742195:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829328.742214:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3878 00000100:00080000:2.0:1787829328.742224:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829328.742230:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829328.742232:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829328.742236:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829328.742248:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280873344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829328.742255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829328.742258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829328.742259:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829328.742261:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3878 00000100:00080000:2.0:1787829328.742265:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829328.742267:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829328.742270:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829328.742281:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280873472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829328.742289:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829328/real 1787829328] req@ffff8ab3f0439680 x1874671280872960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829328.742308:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280872960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829328.742316:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829328.742319:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829328.742322:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280872960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829328.742360:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280873088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829328.742378:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829328.742387:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829328/real 1787829328] req@ffff8ab3f043b0c0 x1874671280873088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829328.742397:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280873088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829328.742403:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829328.742405:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829328.742407:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280873088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829328.742411:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280873216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829328.742422:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829328.742428:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829328/real 1787829328] req@ffff8ab3f043a580 x1874671280873216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829328.742436:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280873216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829328.742446:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829328.742448:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829328.742451:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280873216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829328.742454:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280873344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829328.742464:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829328.742470:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829328/real 1787829328] req@ffff8ab3f0438f00 x1874671280873344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829328.742477:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280873344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829328.742484:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829328.742486:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829328.742488:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280873344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829328.742492:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280873472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829328.742502:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829328.742508:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829328/real 1787829328] req@ffff8ab3f0438b40 x1874671280873472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829328.742515:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280873472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829328.742521:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829328.742523:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829328.742525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280873472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829329.766043:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829330.790043:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829331.814068:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829332.838086:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829333.861978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829333.861986:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829333.861990:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829333.861998:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3883 00000100:00080000:0.0:1787829333.862008:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829333.862014:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829333.862043:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829333.862054:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280873600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829333.862074:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829333.862077:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829333.862079:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829333.862081:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3883 00000100:00080000:0.0:1787829333.862084:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829333.862087:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829333.862095:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829333.862105:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280873728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829333.862114:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829333.862115:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280873600:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829333.862122:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829333.862125:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829333.862129:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3883 00000100:00080000:0.0:1787829333.862131:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829333.862135:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829333.862138:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829333.862147:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280873856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829333.862154:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829333.862157:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829333.862158:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829333.862160:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3883 00000100:00080000:0.0:1787829333.862165:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829333.862167:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829333.862171:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829333.862175:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829333.862180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280873984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829333.862188:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829333.862191:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829333.862193:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829333.862197:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3883 00000100:00080000:0.0:1787829333.862200:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829333.862203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829333.862207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829333.862217:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829333/real 1787829333] req@ffff8ab3f075da40 x1874671280873600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829333.862218:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280874112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829333.862257:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280873600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829333.862266:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829333.862269:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829333.862272:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280873600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829333.862310:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280873728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829333.862328:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829333.862339:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829333/real 1787829333] req@ffff8ab3f075e940 x1874671280873728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829333.862347:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280873728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829333.862353:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829333.862355:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829333.862357:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280873728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829333.862362:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280873856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829333.862372:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829333.862379:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829333/real 1787829333] req@ffff8ab3f075ed00 x1874671280873856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829333.862386:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280873856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829333.862392:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829333.862393:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829333.862396:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280873856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829333.862399:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280873984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829333.862410:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829333.862415:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829333/real 1787829333] req@ffff8ab3f075f840 x1874671280873984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829333.862423:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280873984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829333.862429:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829333.862431:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829333.862433:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280873984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829333.862436:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280874112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829333.862445:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829333.862450:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829333/real 1787829333] req@ffff8ab3f075f480 x1874671280874112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829333.862457:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280874112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829333.862464:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829333.862466:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829333.862468:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280874112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829334.886060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829335.910045:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829336.934078:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829337.958039:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829338.981990:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829338.981999:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829338.982003:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829338.982011:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3888 00000100:00080000:3.0:1787829338.982021:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829338.982028:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829338.982053:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829338.982068:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280874240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829338.982087:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829338.982090:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829338.982092:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829338.982094:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3888 00000100:00080000:3.0:1787829338.982097:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829338.982100:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829338.982108:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829338.982117:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280874368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829338.982122:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280874240:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829338.982131:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829338.982134:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829338.982135:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829338.982139:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3888 00000100:00080000:3.0:1787829338.982142:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829338.982145:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829338.982149:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829338.982158:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280874496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829338.982165:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829338.982168:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829338.982169:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829338.982172:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3888 00000100:00080000:3.0:1787829338.982174:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829338.982177:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829338.982180:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829338.982181:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829338.982192:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280874624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829338.982200:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829338.982202:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829338.982203:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829338.982206:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3888 00000100:00080000:3.0:1787829338.982208:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829338.982211:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829338.982214:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829338.982223:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280874752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829338.982234:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829338/real 1787829338] req@ffff8ab3d0a43c00 x1874671280874240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829338.982250:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280874240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829338.982270:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829338.982273:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829338.982276:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280874240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829338.982295:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280874368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829338.982312:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829338.982321:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829338/real 1787829338] req@ffff8ab3d0a430c0 x1874671280874368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829338.982329:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280874368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829338.982336:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829338.982338:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829338.982341:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280874368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829338.982345:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280874496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829338.982356:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829338.982362:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829338/real 1787829338] req@ffff8ab3d0a41e00 x1874671280874496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829338.982370:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280874496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829338.982375:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829338.982377:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829338.982379:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280874496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829338.982382:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280874624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829338.982392:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829338.982405:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829338/real 1787829338] req@ffff8ab3d0a43480 x1874671280874624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829338.982413:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280874624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829338.982419:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829338.982422:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829338.982424:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280874624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829338.982428:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280874752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829338.982438:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829338.982444:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829338/real 1787829338] req@ffff8ab3d0a421c0 x1874671280874752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829338.982451:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280874752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829338.982461:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829338.982465:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829338.982468:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280874752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829340.006050:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829341.030070:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829342.054031:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829343.078095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829344.101984:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829344.101994:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829344.101999:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829344.102011:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3893 00000100:00080000:0.0:1787829344.102023:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829344.102032:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829344.102035:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829344.102083:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280874880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829344.102108:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829344.102112:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829344.102114:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829344.102119:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3893 00000100:00080000:0.0:1787829344.102124:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829344.102130:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829344.102141:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829344.102144:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280874880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829344.102161:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280875008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829344.102182:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829344.102185:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829344.102187:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829344.102191:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3893 00000100:00080000:0.0:1787829344.102195:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829344.102199:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829344.102203:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829344.102208:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829344.102219:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280875136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829344.102240:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829344.102243:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829344.102244:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829344.102247:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3893 00000100:00080000:0.0:1787829344.102250:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829344.102254:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829344.102260:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829344.102274:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280875264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829344.102275:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829344/real 1787829344] req@ffff8ab3f075e580 x1874671280874880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829344.102291:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829344.102298:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829344.102301:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829344.102306:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3893 00000100:00080000:0.0:1787829344.102309:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829344.102312:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829344.102317:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280874880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829344.102318:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829344.102329:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280875392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829344.102334:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829344.102341:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829344.102346:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280874880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829344.102371:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280875008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829344.102396:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829344.102411:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829344/real 1787829344] req@ffff8ab3f075f480 x1874671280875008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829344.102421:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280875008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829344.102439:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829344.102442:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829344.102444:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280875008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829344.102450:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280875136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829344.102471:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829344.102483:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829344/real 1787829344] req@ffff8ab3f075f840 x1874671280875136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829344.102496:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280875136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829344.102505:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829344.102509:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829344.102512:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280875136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829344.102519:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280875264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829344.102539:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829344.102551:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829344/real 1787829344] req@ffff8ab3f075ed00 x1874671280875264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829344.102565:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280875264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829344.102576:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829344.102579:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829344.102583:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280875264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829344.102590:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280875392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829344.102605:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829344.102612:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829344/real 1787829344] req@ffff8ab3f075e940 x1874671280875392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829344.102622:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280875392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829344.102632:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829344.102635:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829344.102639:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280875392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829345.126039:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829346.150071:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829347.174064:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829348.198095:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829349.222022:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829349.222037:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829349.222042:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829349.222056:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3898 00000100:00080000:3.0:1787829349.222071:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829349.222080:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829349.222086:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829349.222126:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280875520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829349.222153:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829349.222156:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829349.222157:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829349.222160:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3898 00000100:00080000:3.0:1787829349.222162:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829349.222167:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829349.222177:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829349.222201:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280875648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829349.222213:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280875520:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829349.222217:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829349.222226:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829349.222230:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829349.222239:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3898 00000100:00080000:3.0:1787829349.222243:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829349.222246:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829349.222251:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829349.222262:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280875776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829349.222269:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829349.222272:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829349.222274:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829349.222277:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3898 00000100:00080000:3.0:1787829349.222286:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829349.222291:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829349.222295:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829349.222305:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280875904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829349.222312:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829349.222316:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829349.222318:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829349.222325:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3898 00000100:00080000:3.0:1787829349.222328:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829349.222333:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829349.222338:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829349.222354:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280875648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787829349.222355:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280876032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829349.222408:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280875776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829349.222424:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280875904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829349.222448:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280876032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829350.246100:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829351.270076:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829352.294079:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829352.294254:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787829352.294300:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829352.294316:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829352.294354:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829352.294367:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829352.294377:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829352.294388:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787829352.294407:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829349/real 1787829352] req@ffff8ab3d0a43480 x1874671280876032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829352.294436:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280876032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829352.294457:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829352.294463:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829352.294468:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280876032:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829352.294477:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829349/real 1787829352] req@ffff8ab3d0a43840 x1874671280875648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829352.294492:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280875648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829352.294501:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829352.294505:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829352.294509:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280875648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829352.294517:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829349/real 1787829352] req@ffff8ab3d0a412c0 x1874671280875776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829352.294528:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280875776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829352.294536:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829352.294540:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829352.294543:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280875776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829352.294550:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829349/real 1787829352] req@ffff8ab3d0a421c0 x1874671280875904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829352.294561:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280875904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829352.294587:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829352.294592:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829352.294595:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280875904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829352.294603:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829349/real 1787829352] req@ffff8ab3d0a40000 x1874671280875520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829352.294614:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280875520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829352.294622:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829352.294625:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829352.294628:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280875520:192.168.203.124@tcp:250:kworker.0 00010000:00010000:2.0:1787829353.318002:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829354.342041:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829354.342054:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829354.342059:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829354.342089:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3903 00000100:00080000:3.0:1787829354.342102:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829354.342114:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829354.342118:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829354.342174:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280876160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829354.342217:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829354.342220:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829354.342222:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829354.342225:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3903 00000100:00080000:3.0:1787829354.342229:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829354.342233:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829354.342244:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829354.342253:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280876160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829354.342259:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280876288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829354.342290:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829354.342295:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829354.342298:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829354.342313:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3903 00000100:00080000:3.0:1787829354.342327:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829354.342334:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829354.342357:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787829354.342359:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829354.342378:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280876416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829354.342391:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829354.342395:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829354.342398:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829354.342403:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3903 00000100:00080000:3.0:1787829354.342408:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829354.342415:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829354/real 1787829354] req@ffff8ab3d0a41e00 x1874671280876160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829354.342418:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829354.342431:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829354.342448:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280876160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829354.342448:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280876544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829354.342459:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829354.342469:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829354.342479:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829354.342486:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829354.342487:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3903 00000100:00080000:3.0:1787829354.342497:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829354.342498:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787829354.342506:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829354.342509:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280876160:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787829354.342514:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829354.342533:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280876672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829354.342545:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280876288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829354.342586:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829354.342607:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829354/real 1787829354] req@ffff8ab3d0a430c0 x1874671280876288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829354.342624:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280876288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829354.342635:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829354.342639:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829354.342644:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280876288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829354.342651:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280876416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829354.342677:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829354.342691:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829354/real 1787829354] req@ffff8ab3d0a43c00 x1874671280876416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829354.342708:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280876416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829354.342719:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829354.342723:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829354.342727:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280876416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829354.342734:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280876544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829354.342757:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829354.342770:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829354/real 1787829354] req@ffff8ab3d0a40000 x1874671280876544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829354.342792:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280876544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829354.342803:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829354.342806:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829354.342810:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280876544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829354.342830:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280876672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829354.342855:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829354.342869:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829354/real 1787829354] req@ffff8ab3d0a421c0 x1874671280876672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829354.342883:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280876672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829354.342923:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829354.342928:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829354.342932:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280876672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829355.366043:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829356.390071:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829357.414067:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829358.438094:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829359.462001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829359.462016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829359.462021:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829359.462030:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3909 00000100:00080000:2.0:1787829359.462041:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829359.462049:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829359.462068:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829359.462087:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280876800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829359.462108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829359.462113:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829359.462115:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829359.462128:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3909 00000100:00080000:2.0:1787829359.462133:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829359.462138:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829359.462148:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829359.462151:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280876800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829359.462162:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280876928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829359.462172:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829359.462175:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829359.462177:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829359.462180:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3909 00000100:00080000:2.0:1787829359.462184:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829359.462187:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829359.462192:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829359.462201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280877056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829359.462209:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829359.462213:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829359.462214:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829359.462215:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829359.462218:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3909 00000100:00080000:2.0:1787829359.462221:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829359.462225:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829359.462230:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829359.462239:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280877184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829359.462248:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829359.462251:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829359.462253:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829359.462257:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3909 00000100:00100000:1.0:1787829359.462258:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829359/real 1787829359] req@ffff8ab3f0438780 x1874671280876800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829359.462260:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829359.462264:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829359.462268:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829359.462275:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280876800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829359.462276:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280877312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829359.462292:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829359.462296:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829359.462300:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280876800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829359.462323:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280876928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829359.462341:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829359.462351:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829359/real 1787829359] req@ffff8ab3f0438b40 x1874671280876928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829359.462359:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280876928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829359.462365:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829359.462367:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829359.462369:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280876928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829359.462373:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280877056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829359.462384:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829359.462389:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829359/real 1787829359] req@ffff8ab3f0438f00 x1874671280877056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829359.462396:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280877056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829359.462401:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829359.462403:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829359.462406:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280877056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829359.462409:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280877184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829359.462418:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829359.462423:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829359/real 1787829359] req@ffff8ab3f043a580 x1874671280877184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829359.462430:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280877184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829359.462435:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829359.462437:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829359.462439:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280877184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829359.462443:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280877312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829359.462451:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829359.462458:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829359/real 1787829359] req@ffff8ab3f043b0c0 x1874671280877312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829359.462469:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280877312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829359.462478:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829359.462481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829359.462485:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280877312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829360.486062:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829361.510082:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829362.534116:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829363.558127:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829364.582010:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829364.582019:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829364.582024:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829364.582034:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3914 00010000:00010000:2.0:1787829364.582043:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829364.582043:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829364.582052:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829364.582094:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280877440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829364.582115:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829364.582118:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829364.582119:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829364.582122:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3914 00000100:00080000:3.0:1787829364.582125:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829364.582129:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829364.582137:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829364.582149:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280877568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829364.582162:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280877440:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829364.582163:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829364.582180:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829364.582184:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829364.582188:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3914 00000100:00080000:3.0:1787829364.582190:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829364.582195:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829364.582199:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829364.582209:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280877696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829364.582217:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829364.582220:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829364.582221:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829364.582226:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3914 00000100:00080000:3.0:1787829364.582240:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829364.582244:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829364.582249:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829364.582255:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829364.582266:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280877824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829364.582275:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829364.582277:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829364.582280:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829364.582284:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3914 00000100:00080000:3.0:1787829364.582287:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829364.582290:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829364.582294:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829364.582307:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280877952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829364.582320:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829364/real 1787829364] req@ffff8ab3d0a412c0 x1874671280877440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829364.582340:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280877440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829364.582351:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829364.582355:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829364.582359:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280877440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829364.582384:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280877568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829364.582407:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829364.582421:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829364/real 1787829364] req@ffff8ab3d0a43840 x1874671280877568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829364.582436:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280877568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829364.582446:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829364.582450:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829364.582453:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280877568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829364.582458:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280877696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829364.582482:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829364.582491:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829364/real 1787829364] req@ffff8ab3d0a43480 x1874671280877696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829364.582499:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280877696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829364.582512:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829364.582516:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829364.582519:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280877696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829364.582526:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280877824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829364.582549:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829364.582562:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829364/real 1787829364] req@ffff8ab3d0a421c0 x1874671280877824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829364.582575:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280877824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829364.582586:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829364.582590:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829364.582594:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280877824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829364.582601:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280877952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829364.582624:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829364.582637:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829364/real 1787829364] req@ffff8ab3d0a40000 x1874671280877952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829364.582649:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280877952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829364.582670:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829364.582675:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829364.582679:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280877952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829365.606108:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829366.630121:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829367.654073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829368.678070:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829369.701998:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829369.702009:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829369.702014:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829369.702025:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3919 00000100:00080000:3.0:1787829369.702037:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829369.702056:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829369.702115:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280878080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787829369.702123:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829369.702143:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829369.702147:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829369.702149:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829369.702156:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3919 00000100:00080000:3.0:1787829369.702159:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829369.702163:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829369.702174:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829369.702184:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280878080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829369.702186:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280878208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829369.702200:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829369.702202:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829369.702204:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829369.702208:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3919 00000100:00080000:3.0:1787829369.702210:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829369.702213:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829369.702216:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829369.702225:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280878336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829369.702232:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829369.702235:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829369.702236:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829369.702239:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3919 00000100:00080000:3.0:1787829369.702256:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829369.702259:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829369.702262:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829369.702271:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829369.702275:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280878464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829369.702283:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829369.702286:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829369.702288:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829369.702291:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3919 00000100:00080000:3.0:1787829369.702293:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829369.702296:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829369.702299:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829369.702306:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280878592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829369.702343:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829369/real 1787829369] req@ffff8ab3d0a43c00 x1874671280878080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829369.702367:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280878080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829369.702378:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829369.702382:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829369.702386:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280878080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829369.702415:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280878208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829369.702440:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829369.702451:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829369/real 1787829369] req@ffff8ab3d0a430c0 x1874671280878208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829369.702472:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280878208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829369.702478:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829369.702480:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829369.702483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280878208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829369.702488:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280878336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829369.702500:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829369.702507:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829369/real 1787829369] req@ffff8ab3d0a41e00 x1874671280878336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829369.702515:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280878336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829369.702521:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829369.702522:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829369.702524:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280878336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829369.702528:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280878464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829369.702540:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829369.702546:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829369/real 1787829369] req@ffff8ab3d0a40000 x1874671280878464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829369.702554:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280878464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829369.702561:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829369.702562:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829369.702564:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280878464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829369.702568:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280878592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829369.702578:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829369.702583:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829369/real 1787829369] req@ffff8ab3d0a421c0 x1874671280878592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829369.702590:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280878592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829369.702598:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829369.702600:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829369.702602:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280878592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829370.726102:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829371.750081:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829372.774044:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829373.798097:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829374.821985:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829374.821995:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829374.821999:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829374.822022:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3924 00000100:00080000:3.0:1787829374.822035:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829374.822041:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829374.822072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829374.822074:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280878720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829374.822099:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829374.822103:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829374.822105:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829374.822109:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3924 00000100:00080000:3.0:1787829374.822114:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829374.822119:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829374.822130:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829374.822143:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280878720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829374.822149:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280878848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829374.822165:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829374.822169:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829374.822170:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829374.822176:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3924 00000100:00080000:3.0:1787829374.822179:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829374.822184:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829374.822189:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829374.822205:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280878976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829374.822212:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829374.822214:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829374.822216:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829374.822219:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3924 00000100:00080000:3.0:1787829374.822221:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829374.822231:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829374.822231:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829374.822241:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829374.822254:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280879104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829374.822262:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829374.822264:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829374.822266:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829374.822270:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3924 00000100:00080000:3.0:1787829374.822273:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829374.822277:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829374.822281:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829374.822291:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280879232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829374.822309:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829374/real 1787829374] req@ffff8ab3d0a43480 x1874671280878720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829374.822331:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280878720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829374.822341:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829374.822344:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829374.822348:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280878720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829374.822372:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280878848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829374.822392:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829374.822402:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829374/real 1787829374] req@ffff8ab3d0a43840 x1874671280878848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829374.822411:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280878848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829374.822417:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829374.822420:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829374.822422:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280878848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829374.822427:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280878976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829374.822439:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829374.822445:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829374/real 1787829374] req@ffff8ab3d0a412c0 x1874671280878976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829374.822453:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280878976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829374.822467:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829374.822470:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829374.822472:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280878976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829374.822477:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280879104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829374.822489:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829374.822496:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829374/real 1787829374] req@ffff8ab3d0a421c0 x1874671280879104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829374.822504:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280879104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829374.822511:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829374.822512:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829374.822514:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280879104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829374.822518:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280879232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829374.822529:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829374.822534:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829374/real 1787829374] req@ffff8ab3d0a40000 x1874671280879232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829374.822542:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280879232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829374.822549:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829374.822550:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829374.822558:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280879232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829375.846123:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829376.870053:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829377.894078:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829378.918086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829379.942012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829379.942019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829379.942023:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829379.942033:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3929 00000100:00080000:2.0:1787829379.942043:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829379.942050:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829379.942064:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829379.942082:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280879360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829379.942102:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829379.942105:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829379.942107:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829379.942109:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3929 00000100:00080000:2.0:1787829379.942112:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829379.942115:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829379.942123:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829379.942136:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280879360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829379.942136:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280879488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829379.942150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829379.942152:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829379.942153:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829379.942157:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3929 00000100:00080000:2.0:1787829379.942159:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829379.942162:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829379.942166:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829379.942175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280879616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829379.942181:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829379.942185:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829379.942186:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829379.942189:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3929 00000100:00080000:2.0:1787829379.942191:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829379.942196:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829379.942197:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829379.942201:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829379.942212:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280879744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829379.942218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829379.942220:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829379.942221:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829379.942223:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3929 00000100:00080000:2.0:1787829379.942226:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829379.942229:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829379.942233:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829379.942240:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829379/real 1787829379] req@ffff8ab3f0439680 x1874671280879360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829379.942243:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280879872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829379.942273:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280879360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829379.942283:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829379.942286:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829379.942289:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280879360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829379.942307:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280879488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829379.942321:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829379.942330:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829379/real 1787829379] req@ffff8ab3f0439a40 x1874671280879488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829379.942338:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280879488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829379.942348:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829379.942350:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829379.942352:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280879488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829379.942356:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280879616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829379.942365:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829379.942370:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829379/real 1787829379] req@ffff8ab3f043bc00 x1874671280879616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829379.942377:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280879616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829379.942383:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829379.942384:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829379.942386:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280879616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829379.942390:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280879744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829379.942399:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829379.942405:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829379/real 1787829379] req@ffff8ab3f043ad00 x1874671280879744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829379.942412:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280879744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829379.942418:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829379.942420:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829379.942422:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280879744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829379.942426:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280879872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829379.942436:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829379.942441:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829379/real 1787829379] req@ffff8ab3f04383c0 x1874671280879872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829379.942448:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280879872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829379.942454:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829379.942456:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829379.942457:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280879872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829380.966141:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829381.990093:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829383.014070:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829384.038088:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829385.062028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829385.062044:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829385.062048:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829385.062059:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3934 00010000:00010000:2.0:1787829385.062062:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829385.062071:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829385.062080:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829385.062119:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280880000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829385.062146:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829385.062151:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829385.062159:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829385.062164:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3934 00000100:00080000:0.0:1787829385.062168:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829385.062174:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829385.062175:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280880000:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787829385.062183:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829385.062197:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280880128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829385.062211:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829385.062214:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829385.062215:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829385.062219:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3934 00000100:00080000:0.0:1787829385.062223:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829385.062225:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829385.062229:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829385.062232:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829385.062245:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280880256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829385.062253:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829385.062256:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829385.062258:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829385.062260:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3934 00000100:00080000:0.0:1787829385.062262:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829385.062267:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829385.062274:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829385.062282:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829385/real 1787829385] req@ffff8ab3f075da40 x1874671280880000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829385.062297:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280880384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829385.062298:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280880000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829385.062332:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829385.062334:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829385.062338:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787829385.062343:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829385.062345:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280880000:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829385.062347:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829385.062355:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3934 00000100:00080000:0.0:1787829385.062359:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829385.062370:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280880128:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787829385.062371:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829385.062380:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829385.062394:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280880512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829385.062398:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829385.062420:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829385/real 1787829385] req@ffff8ab3f075cb40 x1874671280880128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829385.062437:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280880128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829385.062446:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829385.062450:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829385.062453:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280880128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829385.062458:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280880256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829385.062474:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829385.062481:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829385/real 1787829385] req@ffff8ab3f075f0c0 x1874671280880256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829385.062489:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280880256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829385.062495:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829385.062497:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829385.062499:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280880256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829385.062503:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280880384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829385.062513:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829385.062518:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829385/real 1787829385] req@ffff8ab3f075de00 x1874671280880384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829385.062525:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280880384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829385.062532:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829385.062534:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829385.062536:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280880384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829385.062546:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280880512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829385.062556:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829385.062561:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829385/real 1787829385] req@ffff8ab3f075cf00 x1874671280880512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829385.062568:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280880512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829385.062575:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829385.062576:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829385.062578:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280880512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829386.086099:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829387.110079:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829388.134062:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829389.158118:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829390.182011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829390.182023:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829390.182028:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829390.182039:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3939 00000100:00080000:2.0:1787829390.182051:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829390.182059:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829390.182061:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829390.182106:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280880640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829390.182131:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829390.182134:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829390.182136:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829390.182139:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3939 00000100:00080000:2.0:1787829390.182141:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829390.182145:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829390.182162:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829390.182169:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280880640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829390.182175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280880768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829390.182184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829390.182187:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829390.182189:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829390.182192:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3939 00000100:00080000:2.0:1787829390.182195:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829390.182198:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829390.182201:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829390.182212:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280880896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829390.182219:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829390.182222:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829390.182224:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829390.182227:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3939 00000100:00080000:2.0:1787829390.182232:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829390.182235:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829390.182238:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829390.182244:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829390.182248:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280881024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829390.182255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829390.182258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829390.182260:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829390.182263:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3939 00000100:00080000:2.0:1787829390.182267:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829390.182269:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829390.182272:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829390.182280:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280881152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829390.182299:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829390/real 1787829390] req@ffff8ab3f04392c0 x1874671280880640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829390.182322:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280880640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829390.182335:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829390.182340:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829390.182345:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280880640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829390.182383:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280880768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829390.182409:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829390.182420:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829390/real 1787829390] req@ffff8ab3f043a940 x1874671280880768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829390.182429:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280880768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829390.182435:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829390.182438:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829390.182440:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280880768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829390.182445:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280880896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829390.182456:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829390.182462:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829390/real 1787829390] req@ffff8ab3f043a1c0 x1874671280880896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829390.182470:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280880896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829390.182475:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829390.182477:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829390.182479:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280880896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829390.182482:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280881024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829390.182493:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829390.182500:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829390/real 1787829390] req@ffff8ab3f0439e00 x1874671280881024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829390.182508:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280881024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829390.182515:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829390.182517:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829390.182519:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280881024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829390.182522:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280881152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829390.182534:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829390.182545:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829390/real 1787829390] req@ffff8ab3f043b480 x1874671280881152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829390.182558:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280881152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829390.182568:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829390.182572:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829390.182576:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280881152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829391.206131:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829392.230042:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829393.254068:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829394.278069:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829395.301990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829395.301999:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829395.302003:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829395.302011:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3944 00000100:00080000:2.0:1787829395.302020:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829395.302027:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829395.302052:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829395.302069:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280881280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829395.302090:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829395.302093:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829395.302094:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829395.302097:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3944 00000100:00080000:2.0:1787829395.302100:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829395.302102:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829395.302110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829395.302126:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280881408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829395.302131:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280881280:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829395.302144:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829395.302148:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829395.302150:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829395.302156:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3944 00000100:00080000:2.0:1787829395.302161:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829395.302166:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829395.302172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829395.302188:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280881536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829395.302189:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829395.302201:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829395.302204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829395.302207:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829395.302211:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3944 00000100:00080000:2.0:1787829395.302214:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829395.302218:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829395.302230:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829395.302231:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829395/real 1787829395] req@ffff8ab3f043b840 x1874671280881280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829395.302243:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280881664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829395.302249:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280881280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829395.302257:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829395.302264:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829395.302266:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829395.302266:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829395.302274:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787829395.302276:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3944 00000100:00080000:2.0:1787829395.302281:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829395.302283:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280881280:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829395.302287:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829395.302293:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829395.302300:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280881792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829395.302308:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280881408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829395.302327:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829395.302337:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829395/real 1787829395] req@ffff8ab3f043b480 x1874671280881408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829395.302345:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280881408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829395.302351:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829395.302353:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829395.302356:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280881408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829395.302360:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280881536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829395.302370:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829395.302376:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829395/real 1787829395] req@ffff8ab3f0439e00 x1874671280881536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829395.302383:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280881536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829395.302388:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829395.302391:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829395.302395:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280881536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829395.302401:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280881664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829395.302420:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829395.302431:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829395/real 1787829395] req@ffff8ab3f043a1c0 x1874671280881664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829395.302444:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280881664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829395.302454:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829395.302456:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829395.302459:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280881664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829395.302470:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280881792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829395.302486:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829395.302493:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829395/real 1787829395] req@ffff8ab3f043a940 x1874671280881792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829395.302500:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280881792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829395.302506:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829395.302508:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829395.302510:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280881792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829396.326086:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829397.350091:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829398.374027:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829399.398084:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829400.421981:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829400.421990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829400.421994:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829400.422003:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3950 00000100:00080000:2.0:1787829400.422012:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829400.422019:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829400.422037:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829400.422054:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280881920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829400.422079:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829400.422082:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829400.422083:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829400.422086:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3950 00000100:00080000:2.0:1787829400.422088:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829400.422092:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829400.422100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829400.422110:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280881920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829400.422113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280882048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829400.422124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829400.422126:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829400.422127:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829400.422130:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3950 00000100:00080000:2.0:1787829400.422132:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829400.422135:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829400.422138:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829400.422147:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280882176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829400.422152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829400.422154:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829400.422156:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829400.422159:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3950 00000100:00080000:2.0:1787829400.422162:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829400.422165:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829400.422168:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829400.422170:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829400.422179:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280882304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829400.422188:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829400.422190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829400.422191:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829400.422194:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3950 00000100:00080000:2.0:1787829400.422196:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829400.422199:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829400.422202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829400.422215:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829400/real 1787829400] req@ffff8ab3f04392c0 x1874671280881920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829400.422216:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280882432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829400.422251:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280881920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829400.422260:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829400.422263:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829400.422266:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280881920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829400.422285:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280882048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829400.422300:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829400.422309:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829400/real 1787829400] req@ffff8ab3f04383c0 x1874671280882048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829400.422317:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280882048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829400.422323:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829400.422325:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829400.422327:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280882048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829400.422330:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280882176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829400.422340:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829400.422346:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829400/real 1787829400] req@ffff8ab3f043ad00 x1874671280882176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829400.422353:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280882176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829400.422362:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829400.422364:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829400.422366:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280882176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829400.422370:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280882304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829400.422379:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829400.422384:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829400/real 1787829400] req@ffff8ab3f043bc00 x1874671280882304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829400.422391:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280882304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829400.422397:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829400.422399:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829400.422401:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280882304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829400.422404:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280882432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829400.422413:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829400.422418:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829400/real 1787829400] req@ffff8ab3f0439a40 x1874671280882432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829400.422425:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280882432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829400.422431:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829400.422433:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829400.422435:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280882432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829401.446044:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829402.470013:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829403.494036:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829404.518110:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829405.542028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829405.542043:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1787829405.542048:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829405.542050:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829405.542064:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3955 00000100:00080000:0.0:1787829405.542078:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829405.542087:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829405.542152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280882560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829405.542187:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829405.542192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829405.542194:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829405.542198:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3955 00000100:00080000:0.0:1787829405.542203:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829405.542208:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829405.542220:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829405.542228:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280882560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829405.542241:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280882688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829405.542255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829405.542260:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829405.542262:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829405.542267:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3955 00000100:00080000:0.0:1787829405.542284:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829405.542290:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829405.542294:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787829405.542307:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829405.542328:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280882816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829405.542340:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829405.542344:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829405.542346:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829405.542352:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3955 00000100:00100000:1.0:1787829405.542357:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829405/real 1787829405] req@ffff8ab3f075c000 x1874671280882560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829405.542359:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829405.542371:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829405.542380:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829405.542387:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280882560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829405.542397:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280882944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829405.542403:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829405.542412:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787829405.542413:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829405.542417:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280882560:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829405.542424:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829405.542428:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829405.542434:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3955 00000100:00080000:0.0:1787829405.542439:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829405.542444:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829405.542450:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829405.542454:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280882688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787829405.542467:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280883072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829405.542487:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829405.542502:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829405/real 1787829405] req@ffff8ab3f075cf00 x1874671280882688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829405.542513:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280882688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829405.542522:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829405.542526:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829405.542530:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280882688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829405.542537:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280882816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829405.542558:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829405.542570:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829405/real 1787829405] req@ffff8ab3f075de00 x1874671280882816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829405.542587:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280882816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829405.542594:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829405.542596:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829405.542599:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280882816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829405.542603:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280882944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829405.542618:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829405.542625:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829405/real 1787829405] req@ffff8ab3f075f0c0 x1874671280882944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829405.542633:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280882944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829405.542640:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829405.542642:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829405.542644:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280882944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829405.542657:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280883072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829405.542667:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829405.542673:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829405/real 1787829405] req@ffff8ab3f075cb40 x1874671280883072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829405.542680:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280883072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829405.542688:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829405.542690:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829405.542692:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280883072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829406.566082:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829407.590011:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829408.614093:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829409.638043:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829410.661995:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829410.662007:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829410.662012:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829410.662028:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3960 00000100:00080000:0.0:1787829410.662042:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829410.662049:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829410.662071:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829410.662084:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280883200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829410.662107:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829410.662112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829410.662114:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829410.662118:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3960 00000100:00080000:0.0:1787829410.662122:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829410.662127:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829410.662143:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829410.662157:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280883200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829410.662160:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280883328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829410.662175:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829410.662178:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829410.662181:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829410.662185:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3960 00000100:00080000:0.0:1787829410.662190:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829410.662195:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829410.662201:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829410.662213:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280883456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829410.662223:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829410.662226:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829410.662231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829410.662234:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829410.662240:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3960 00000100:00080000:0.0:1787829410.662244:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829410.662250:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829410.662256:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829410.662269:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280883584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829410.662278:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829410/real 1787829410] req@ffff8ab3f075da40 x1874671280883200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829410.662284:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829410.662294:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829410.662298:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829410.662306:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3960 00000100:00100000:1.0:1787829410.662311:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280883200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829410.662313:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829410.662324:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787829410.662332:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829410.662333:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787829410.662341:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829410.662347:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280883200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829410.662352:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280883712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829410.662377:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280883328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829410.662405:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829410.662420:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829410/real 1787829410] req@ffff8ab3f075e940 x1874671280883328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829410.662435:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280883328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829410.662441:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829410.662444:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829410.662446:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280883328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829410.662451:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280883456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829410.662466:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829410.662473:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829410/real 1787829410] req@ffff8ab3f075ed00 x1874671280883456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829410.662481:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280883456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829410.662486:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829410.662488:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829410.662491:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280883456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829410.662494:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280883584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829410.662508:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829410.662518:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829410/real 1787829410] req@ffff8ab3f075f840 x1874671280883584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829410.662530:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280883584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829410.662540:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829410.662544:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829410.662558:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280883584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829410.662563:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280883712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829410.662579:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829410.662586:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829410/real 1787829410] req@ffff8ab3f075f480 x1874671280883712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829410.662594:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280883712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829410.662601:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829410.662603:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829410.662605:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280883712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829411.686053:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829412.710067:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829413.734027:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829414.758076:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829415.781955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829415.781963:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829415.781967:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829415.781976:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3965 00000100:00080000:0.0:1787829415.781986:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829415.781993:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829415.782027:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829415.782032:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280883840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829415.782059:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829415.782062:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829415.782064:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829415.782067:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3965 00000100:00080000:0.0:1787829415.782071:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829415.782076:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829415.782085:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829415.782093:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280883840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829415.782100:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280883968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829415.782110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829415.782113:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829415.782114:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829415.782117:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3965 00000100:00080000:0.0:1787829415.782120:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829415.782122:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829415.782126:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829415.782134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280884096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829415.782141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829415.782144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829415.782145:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829415.782147:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3965 00000100:00080000:0.0:1787829415.782150:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829415.782154:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829415.782159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829415.782170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280884224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829415.782187:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829415.782191:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829415.782194:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829415.782199:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3965 00000100:00080000:0.0:1787829415.782204:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829415.782208:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829415.782214:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829415.782215:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280883968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787829415.782229:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280884352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829415.782253:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280884096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829415.782280:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280884224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829415.782307:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280884352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829416.806091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829417.830063:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829418.854208:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829418.854248:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787829418.854302:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829418.854325:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829418.854366:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829418.854380:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829418.854394:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829418.854416:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787829418.854418:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829415/real 1787829418] req@ffff8ab3f075f480 x1874671280883968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829418.854463:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280883968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829418.854477:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829418.854482:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829418.854486:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280883968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829418.854496:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829415/real 1787829418] req@ffff8ab3f075f840 x1874671280884096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829418.854518:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280884096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829418.854528:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829418.854531:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829418.854535:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280884096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829418.854542:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829415/real 1787829418] req@ffff8ab3f075ed00 x1874671280884224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829418.854555:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280884224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829418.854564:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829418.854568:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829418.854572:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280884224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829418.854579:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829415/real 1787829418] req@ffff8ab3f075e580 x1874671280883840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829418.854592:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280883840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829418.854600:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829418.854603:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829418.854606:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280883840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829418.854635:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829415/real 1787829418] req@ffff8ab3f075e940 x1874671280884352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829418.854647:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280884352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829418.854656:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829418.854660:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829418.854663:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280884352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829419.878042:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829420.901989:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829420.902000:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829420.902005:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829420.902016:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3970 00000100:00080000:2.0:1787829420.902028:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829420.902036:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829420.902038:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829420.902071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280884480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829420.902098:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829420.902102:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829420.902105:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829420.902111:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3970 00000100:00080000:2.0:1787829420.902116:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829420.902122:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829420.902127:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280884480:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787829420.902138:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829420.902159:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280884608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829420.902171:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829420.902175:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829420.902177:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829420.902182:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3970 00000100:00080000:2.0:1787829420.902185:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829420.902188:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829420.902193:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829420.902196:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829420.902220:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280884736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829420.902232:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829420.902236:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829420.902238:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829420.902242:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3970 00000100:00080000:2.0:1787829420.902246:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829420.902249:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829420/real 1787829420] req@ffff8ab3f0439680 x1874671280884480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829420.902255:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829420.902264:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829420.902280:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280884480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829420.902282:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280884864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829420.902297:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829420.902305:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829420.902309:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829420.902317:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829420.902317:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3970 00000100:00080000:2.0:1787829420.902326:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829420.902329:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787829420.902335:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829420.902340:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280884480:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787829420.902344:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829420.902362:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280884992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829420.902372:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280884608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829420.902403:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829420.902417:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829420/real 1787829420] req@ffff8ab3f043b0c0 x1874671280884608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829420.902430:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280884608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829420.902444:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829420.902447:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829420.902451:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280884608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829420.902458:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280884736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829420.902478:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829420.902490:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829420/real 1787829420] req@ffff8ab3f043a580 x1874671280884736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829420.902503:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280884736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829420.902512:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829420.902515:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829420.902518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280884736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829420.902525:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280884864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829420.902545:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829420.902557:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829420/real 1787829420] req@ffff8ab3f0438f00 x1874671280884864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829420.902568:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280884864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829420.902579:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829420.902582:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829420.902585:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280884864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829420.902601:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280884992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829420.902620:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829420.902632:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829420/real 1787829420] req@ffff8ab3f0438b40 x1874671280884992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829420.902644:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280884992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829420.902654:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829420.902657:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829420.902661:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280884992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829421.926061:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829422.950127:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829423.974089:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829424.998125:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829426.022029:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829426.022038:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829426.022043:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829426.022051:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3975 00000100:00080000:0.0:1787829426.022061:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829426.022069:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829426.022079:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829426.022102:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280885120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829426.022126:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829426.022131:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829426.022133:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829426.022137:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3975 00000100:00080000:0.0:1787829426.022141:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829426.022147:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829426.022157:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829426.022166:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280885120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829426.022173:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280885248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829426.022183:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829426.022186:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829426.022187:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829426.022190:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3975 00000100:00080000:0.0:1787829426.022193:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829426.022196:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829426.022200:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829426.022211:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280885376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829426.022221:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829426.022226:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787829426.022228:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829426.022229:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829426.022237:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3975 00000100:00080000:0.0:1787829426.022242:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829426.022248:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829426.022254:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829426.022267:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280885504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829426.022274:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829426.022277:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829426.022278:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829426.022282:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3975 00000100:00080000:0.0:1787829426.022285:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829426.022290:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829426.022293:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829426.022299:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829426/real 1787829426] req@ffff8ab3f075da40 x1874671280885120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829426.022301:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280885632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829426.022335:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280885120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829426.022347:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829426.022351:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829426.022356:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280885120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829426.022378:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280885248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829426.022402:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829426.022417:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829426/real 1787829426] req@ffff8ab3f075cb40 x1874671280885248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829426.022429:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280885248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829426.022437:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829426.022440:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829426.022444:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280885248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829426.022450:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280885376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829426.022469:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829426.022480:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829426/real 1787829426] req@ffff8ab3f075f0c0 x1874671280885376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829426.022491:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280885376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829426.022498:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829426.022501:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829426.022504:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280885376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829426.022509:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280885504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829426.022527:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829426.022536:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829426/real 1787829426] req@ffff8ab3f075de00 x1874671280885504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829426.022547:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280885504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829426.022557:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829426.022560:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829426.022563:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280885504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829426.022569:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280885632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829426.022586:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829426.022598:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829426/real 1787829426] req@ffff8ab3f075cf00 x1874671280885632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829426.022610:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280885632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829426.022620:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829426.022624:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829426.022627:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280885632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829427.046125:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829428.070081:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829429.094045:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829430.118079:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829431.141980:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829431.141989:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829431.141993:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829431.142002:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3980 00000100:00080000:0.0:1787829431.142011:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829431.142017:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829431.142030:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829431.142051:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280885760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829431.142072:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829431.142083:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829431.142085:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829431.142089:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3980 00000100:00080000:0.0:1787829431.142092:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829431.142095:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829431.142103:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829431.142112:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280885760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829431.142114:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280885888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829431.142125:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829431.142128:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829431.142129:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829431.142133:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3980 00000100:00080000:0.0:1787829431.142135:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829431.142137:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829431.142140:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829431.142149:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280886016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829431.142155:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829431.142159:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829431.142160:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829431.142162:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3980 00000100:00080000:0.0:1787829431.142165:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829431.142167:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829431.142173:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829431.142173:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829431.142184:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280886144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829431.142192:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829431.142194:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829431.142196:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829431.142199:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3980 00000100:00080000:0.0:1787829431.142201:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829431.142203:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829431.142206:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829431.142213:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280886272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829431.142236:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829431/real 1787829431] req@ffff8ab3f075c000 x1874671280885760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829431.142251:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280885760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829431.142259:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829431.142262:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829431.142265:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280885760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829431.142285:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280885888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829431.142303:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829431.142312:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829431/real 1787829431] req@ffff8ab3f075cf00 x1874671280885888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829431.142320:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280885888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829431.142325:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829431.142327:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829431.142330:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280885888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829431.142333:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280886016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829431.142343:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829431.142349:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829431/real 1787829431] req@ffff8ab3f075de00 x1874671280886016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829431.142356:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280886016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829431.142361:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829431.142363:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829431.142365:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280886016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829431.142368:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280886144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829431.142378:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829431.142384:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829431/real 1787829431] req@ffff8ab3f075f0c0 x1874671280886144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829431.142391:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280886144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829431.142398:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829431.142399:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829431.142401:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280886144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829431.142405:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280886272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829431.142414:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829431.142419:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829431/real 1787829431] req@ffff8ab3f075cb40 x1874671280886272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829431.142426:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280886272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829431.142432:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829431.142434:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829431.142436:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280886272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829432.166071:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829433.190059:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829434.214048:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829435.238060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829436.261993:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829436.262001:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829436.262005:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829436.262013:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3985 00000100:00080000:2.0:1787829436.262022:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829436.262029:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829436.262037:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829436.262061:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280886400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829436.262081:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829436.262084:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829436.262085:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829436.262089:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3985 00000100:00080000:2.0:1787829436.262092:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829436.262096:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829436.262106:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829436.262124:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280886400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829436.262125:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280886528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829436.262141:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829436.262145:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829436.262146:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829436.262150:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3985 00000100:00080000:2.0:1787829436.262154:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829436.262158:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829436.262163:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829436.262172:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280886656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829436.262181:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829436.262183:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829436.262184:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829436.262190:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829436.262191:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3985 00000100:00080000:2.0:1787829436.262197:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829436.262200:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829436.262205:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829436.262216:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280886784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829436.262226:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829436.262230:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829436.262232:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829436.262237:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3985 00000100:00080000:2.0:1787829436.262241:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829436.262243:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829436/real 1787829436] req@ffff8ab3f0438780 x1874671280886400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829436.262249:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829436.262257:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829436.262273:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280886400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829436.262275:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280886912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829436.262308:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829436.262312:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829436.262317:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280886400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829436.262336:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280886528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829436.262355:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829436.262365:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829436/real 1787829436] req@ffff8ab3f0438b40 x1874671280886528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829436.262373:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280886528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829436.262379:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829436.262381:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829436.262383:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280886528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829436.262386:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280886656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829436.262396:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829436.262402:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829436/real 1787829436] req@ffff8ab3f0438f00 x1874671280886656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829436.262409:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280886656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829436.262414:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829436.262416:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829436.262418:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280886656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829436.262422:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280886784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829436.262438:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829436.262448:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829436/real 1787829436] req@ffff8ab3f043a580 x1874671280886784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829436.262461:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280886784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829436.262472:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829436.262475:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829436.262479:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280886784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829436.262484:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280886912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829436.262498:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829436.262506:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829436/real 1787829436] req@ffff8ab3f043b0c0 x1874671280886912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829436.262513:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280886912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829436.262520:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829436.262522:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829436.262524:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280886912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829437.286026:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829438.310060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829439.334064:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829440.358193:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829441.382030:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829441.382038:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829441.382042:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829441.382051:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3991 00000100:00080000:2.0:1787829441.382064:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829441.382072:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829441.382099:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829441.382112:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280887040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829441.382137:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829441.382140:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829441.382142:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829441.382145:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3991 00000100:00080000:2.0:1787829441.382148:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829441.382151:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829441.382160:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829441.382171:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280887168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829441.382180:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829441.382182:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280887040:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829441.382183:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829441.382187:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829441.382194:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3991 00000100:00080000:2.0:1787829441.382196:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829441.382199:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829441.382202:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829441.382211:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280887296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829441.382218:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829441.382221:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829441.382222:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829441.382224:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3991 00000100:00080000:2.0:1787829441.382241:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829441.382244:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829441.382247:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829441.382253:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829441.382256:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280887424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829441.382264:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829441.382266:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829441.382268:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829441.382270:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3991 00000100:00080000:2.0:1787829441.382274:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829441.382276:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829441.382279:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829441.382287:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280887552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829441.382321:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829441/real 1787829441] req@ffff8ab3f0439680 x1874671280887040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829441.382348:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280887040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829441.382358:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829441.382362:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829441.382367:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280887040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829441.382391:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280887168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829441.382417:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829441.382432:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829441/real 1787829441] req@ffff8ab3f0439a40 x1874671280887168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829441.382445:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280887168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829441.382454:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829441.382457:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829441.382460:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280887168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829441.382466:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280887296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829441.382485:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829441.382496:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829441/real 1787829441] req@ffff8ab3f043bc00 x1874671280887296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829441.382509:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280887296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829441.382524:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829441.382527:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829441.382531:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280887296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829441.382537:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280887424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829441.382575:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829441.382588:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829441/real 1787829441] req@ffff8ab3f043ad00 x1874671280887424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829441.382600:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280887424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829441.382610:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829441.382614:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829441.382617:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280887424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829441.382623:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280887552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829441.382642:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829441.382655:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829441/real 1787829441] req@ffff8ab3f04383c0 x1874671280887552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829441.382667:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280887552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829441.382676:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829441.382680:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829441.382684:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280887552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829442.406060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829443.430103:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829444.454111:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829445.478067:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829446.501988:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829446.502000:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829446.502006:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829446.502017:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 3996 00000100:00080000:0.0:1787829446.502028:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829446.502037:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829446.502072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829446.502073:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280887680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829446.502101:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829446.502104:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829446.502105:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829446.502109:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3996 00000100:00080000:0.0:1787829446.502113:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829446.502117:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829446.502126:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829446.502135:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280887808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829446.502148:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829446.502150:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280887680:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829446.502154:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829446.502159:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829446.502168:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3996 00000100:00080000:0.0:1787829446.502170:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829446.502173:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829446.502177:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829446.502189:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280887936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829446.502196:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829446.502199:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829446.502201:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829446.502203:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3996 00000100:00080000:0.0:1787829446.502206:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829446.502210:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829446.502214:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829446.502223:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280888064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829446.502229:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829446.502232:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829446.502233:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829446.502236:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 3996 00000100:00080000:0.0:1787829446.502238:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829446.502240:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829446.502244:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829446.502252:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280888192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829446.502255:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829446.502332:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829446/real 1787829446] req@ffff8ab3f075da40 x1874671280887680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829446.502357:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280887680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829446.502371:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829446.502376:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829446.502381:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280887680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829446.502412:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280887808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829446.502433:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829446.502444:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829446/real 1787829446] req@ffff8ab3f075e940 x1874671280887808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829446.502453:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280887808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829446.502459:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829446.502462:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829446.502464:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280887808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829446.502468:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280887936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829446.502480:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829446.502487:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829446/real 1787829446] req@ffff8ab3f075e580 x1874671280887936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829446.502495:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280887936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829446.502501:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829446.502503:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829446.502506:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280887936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829446.502525:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280888064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829446.502549:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829446.502562:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829446/real 1787829446] req@ffff8ab3f075ed00 x1874671280888064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829446.502576:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280888064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829446.502587:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829446.502591:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829446.502595:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280888064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829446.502602:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280888192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829446.502619:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829446.502628:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829446/real 1787829446] req@ffff8ab3f075f840 x1874671280888192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829446.502636:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280888192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829446.502643:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829446.502646:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829446.502648:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280888192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829447.526108:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829448.550092:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829449.574106:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829450.598041:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829451.622035:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829451.622044:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829451.622048:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829451.622057:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4001 00000100:00080000:2.0:1787829451.622067:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829451.622074:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829451.622095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829451.622110:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280888320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829451.622132:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829451.622135:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829451.622136:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829451.622139:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4001 00000100:00080000:2.0:1787829451.622141:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829451.622143:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829451.622151:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829451.622163:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280888448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829451.622171:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829451.622173:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829451.622175:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829451.622179:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4001 00000100:00080000:2.0:1787829451.622180:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829451.622183:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829451.622187:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829451.622197:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280888320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829451.622198:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280888576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829451.622220:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829451.622224:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829451.622226:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829451.622232:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4001 00000100:00080000:2.0:1787829451.622236:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829451.622242:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829451.622247:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829451.622262:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280888704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829451.622270:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829451.622273:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829451.622274:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829451.622285:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4001 00000800:00000100:1.0:1787829451.622293:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829451.622294:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829451.622306:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829451.622314:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829451.622331:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280888832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829451.622372:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829451/real 1787829451] req@ffff8ab3f04392c0 x1874671280888320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829451.622390:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280888320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829451.622401:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829451.622404:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829451.622408:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280888320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829451.622416:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280888448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829451.622435:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829451.622444:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829451/real 1787829451] req@ffff8ab3f043a940 x1874671280888448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829451.622453:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280888448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829451.622459:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829451.622462:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829451.622464:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280888448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829451.622508:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280888576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829451.622525:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829451.622533:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829451/real 1787829451] req@ffff8ab3f043a1c0 x1874671280888576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829451.622541:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280888576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829451.622547:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829451.622550:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829451.622552:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280888576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829451.622556:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280888704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829451.622567:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829451.622573:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829451/real 1787829451] req@ffff8ab3f0439e00 x1874671280888704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829451.622580:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280888704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829451.622588:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829451.622590:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829451.622592:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280888704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829451.622596:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280888832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829451.622606:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829451.622612:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829451/real 1787829451] req@ffff8ab3f043b480 x1874671280888832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829451.622620:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280888832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829451.622627:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829451.622629:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829451.622631:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280888832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829452.646042:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829453.670109:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829454.694089:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829455.718068:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829456.742006:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1787829456.742011:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829456.742015:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829456.742020:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829456.742029:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4006 00000100:00080000:0.0:1787829456.742038:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829456.742045:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829456.742078:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280888960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829456.742100:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829456.742105:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829456.742106:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829456.742109:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4006 00000100:00080000:0.0:1787829456.742112:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829456.742115:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829456.742123:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829456.742133:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280889088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829456.742143:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829456.742144:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280888960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829456.742146:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829456.742150:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829456.742159:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4006 00000100:00080000:0.0:1787829456.742162:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829456.742166:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829456.742172:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829456.742186:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280889216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829456.742197:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829456.742201:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829456.742203:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829456.742208:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4006 00000100:00080000:0.0:1787829456.742213:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829456.742218:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829456.742223:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829456.742230:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829456.742241:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280889344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829456.742249:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829456.742251:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829456.742253:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829456.742256:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4006 00000100:00080000:0.0:1787829456.742259:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829456.742262:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829456.742266:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829456.742278:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280889472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829456.742279:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829456/real 1787829456] req@ffff8ab3f075f480 x1874671280888960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829456.742311:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280888960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829456.742322:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829456.742326:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829456.742330:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280888960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829456.742353:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280889088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829456.742378:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829456.742392:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829456/real 1787829456] req@ffff8ab3f075f840 x1874671280889088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829456.742405:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280889088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829456.742420:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829456.742423:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829456.742427:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280889088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829456.742434:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280889216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829456.742471:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829456.742483:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829456/real 1787829456] req@ffff8ab3f075ed00 x1874671280889216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829456.742495:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280889216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829456.742504:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829456.742507:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829456.742511:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280889216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829456.742517:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280889344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829456.742538:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829456.742552:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829456/real 1787829456] req@ffff8ab3f075e580 x1874671280889344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829456.742564:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280889344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829456.742587:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829456.742592:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829456.742596:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280889344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829456.742603:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280889472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829456.742624:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829456.742637:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829456/real 1787829456] req@ffff8ab3f075e940 x1874671280889472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829456.742651:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280889472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829456.742661:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829456.742664:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829456.742667:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280889472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829457.766088:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829458.790067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829459.814087:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829460.838067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829461.862043:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829461.862052:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829461.862057:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829461.862067:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4011 00000100:00080000:0.0:1787829461.862077:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829461.862085:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829461.862126:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280889600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787829461.862153:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829461.862162:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829461.862166:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829461.862168:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829461.862173:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4011 00000100:00080000:0.0:1787829461.862179:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829461.862182:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829461.862192:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829461.862203:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280889728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829461.862214:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829461.862220:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829461.862223:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829461.862223:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280889600:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829461.862230:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4011 00000100:00080000:0.0:1787829461.862243:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829461.862247:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829461.862251:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829461.862259:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280889856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829461.862266:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829461.862268:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829461.862269:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829461.862272:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4011 00000100:00080000:0.0:1787829461.862274:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829461.862277:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829461.862280:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829461.862286:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280889984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829461.862300:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829461.862304:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829461.862306:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829461.862311:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4011 00000100:00080000:0.0:1787829461.862315:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829461.862320:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829461.862329:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787829461.862330:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829461.862347:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280890112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829461.862424:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829461/real 1787829461] req@ffff8ab3f075da40 x1874671280889600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829461.862448:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280889600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829461.862461:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829461.862466:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829461.862470:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280889600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829461.862502:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280889728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829461.862530:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829461.862546:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829461/real 1787829461] req@ffff8ab3f075cb40 x1874671280889728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829461.862560:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280889728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829461.862569:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829461.862573:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829461.862577:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280889728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829461.862584:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280889856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829461.862604:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829461.862616:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829461/real 1787829461] req@ffff8ab3f075f0c0 x1874671280889856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829461.862629:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280889856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829461.862638:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829461.862641:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829461.862644:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280889856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829461.862651:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280889984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829461.862673:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829461.862685:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829461/real 1787829461] req@ffff8ab3f075de00 x1874671280889984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829461.862697:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280889984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829461.862709:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829461.862712:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829461.862716:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280889984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829461.862723:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280890112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829461.862744:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829461.862757:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829461/real 1787829461] req@ffff8ab3f075cf00 x1874671280890112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829461.862769:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280890112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829461.862780:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829461.862784:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829461.862787:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280890112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829462.886161:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829463.910098:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829464.934094:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829465.958063:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829466.982007:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829466.982018:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829466.982023:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829466.982034:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4016 00000100:00080000:0.0:1787829466.982044:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829466.982052:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829466.982073:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829466.982101:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280890240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829466.982124:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829466.982127:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829466.982128:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829466.982131:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4016 00000100:00080000:0.0:1787829466.982134:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829466.982137:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829466.982146:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829466.982159:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280890368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829466.982174:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829466.982174:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280890240:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829466.982183:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829466.982185:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829466.982189:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4016 00000100:00080000:0.0:1787829466.982191:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829466.982193:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829466.982197:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829466.982207:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280890496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829466.982214:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829466.982217:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829466.982218:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829466.982221:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4016 00000100:00080000:0.0:1787829466.982236:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829466.982239:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829466.982242:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829466.982249:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829466.982252:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280890624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829466.982265:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829466.982268:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829466.982269:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829466.982273:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4016 00000100:00080000:0.0:1787829466.982278:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829466.982281:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829466.982284:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829466.982292:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280890752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829466.982328:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829466/real 1787829466] req@ffff8ab3f075c000 x1874671280890240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829466.982355:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280890240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829466.982381:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829466.982387:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829466.982391:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280890240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829466.982423:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280890368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829466.982451:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829466.982467:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829466/real 1787829466] req@ffff8ab3f075cf00 x1874671280890368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829466.982482:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280890368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829466.982492:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829466.982496:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829466.982500:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280890368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829466.982505:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280890496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829466.982522:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829466.982530:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829466/real 1787829466] req@ffff8ab3f075de00 x1874671280890496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829466.982538:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280890496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829466.982545:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829466.982547:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829466.982563:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280890496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829466.982567:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280890624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829466.982581:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829466.982589:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829466/real 1787829466] req@ffff8ab3f075f0c0 x1874671280890624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829466.982604:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280890624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829466.982611:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829466.982614:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829466.982616:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280890624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829466.982620:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280890752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829466.982631:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829466.982643:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829466/real 1787829466] req@ffff8ab3f075cb40 x1874671280890752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829466.982651:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280890752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829466.982659:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829466.982661:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829466.982663:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280890752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829468.006039:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829469.030081:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829470.054058:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829471.078083:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829472.101989:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829472.102000:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829472.102004:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829472.102015:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4021 00000100:00080000:0.0:1787829472.102026:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829472.102035:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829472.102052:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829472.102072:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280890880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829472.102102:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829472.102106:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829472.102109:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829472.102113:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4021 00000100:00080000:0.0:1787829472.102117:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829472.102123:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829472.102134:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829472.102146:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280890880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829472.102149:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280891008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829472.102164:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829472.102168:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829472.102171:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829472.102176:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4021 00000100:00080000:0.0:1787829472.102181:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829472.102186:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829472.102191:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829472.102203:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280891136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829472.102213:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829472.102220:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787829472.102220:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829472.102224:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829472.102230:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4021 00000100:00080000:0.0:1787829472.102235:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829472.102240:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829472.102246:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829472.102259:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280891264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829472.102271:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829472/real 1787829472] req@ffff8ab3f075da40 x1874671280890880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829472.102272:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829472.102281:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829472.102287:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829472.102294:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4021 00000100:00100000:1.0:1787829472.102299:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280890880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829472.102300:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829472.102310:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829472.102319:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787829472.102323:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829472.102330:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787829472.102348:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280891392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829472.102351:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280890880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829472.102384:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280891008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829472.102404:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829472.102414:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829472/real 1787829472] req@ffff8ab3f075e940 x1874671280891008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829472.102423:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280891008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829472.102429:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829472.102431:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829472.102433:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280891008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829472.102437:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280891136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829472.102447:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829472.102453:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829472/real 1787829472] req@ffff8ab3f075e580 x1874671280891136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829472.102460:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280891136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829472.102465:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829472.102467:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829472.102470:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280891136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829472.102474:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280891264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829472.102483:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829472.102489:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829472/real 1787829472] req@ffff8ab3f075ed00 x1874671280891264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829472.102496:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280891264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829472.102502:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829472.102504:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829472.102506:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280891264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829472.102509:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280891392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829472.102518:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829472.102524:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829472/real 1787829472] req@ffff8ab3f075f840 x1874671280891392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829472.102531:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280891392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829472.102536:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829472.102538:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829472.102540:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280891392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829473.126081:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829474.150051:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829475.174109:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829476.198092:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829477.222019:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829477.222027:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829477.222031:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829477.222041:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4026 00000100:00080000:2.0:1787829477.222050:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829477.222057:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829477.222061:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829477.222094:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280891520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829477.222115:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829477.222118:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829477.222119:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829477.222122:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4026 00000100:00080000:2.0:1787829477.222125:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829477.222129:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829477.222137:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829477.222147:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280891520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829477.222152:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280891648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829477.222165:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829477.222169:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829477.222171:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829477.222176:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4026 00000100:00080000:2.0:1787829477.222199:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829477.222204:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829477.222210:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829477.222214:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829477.222231:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280891776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829477.222242:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829477.222245:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829477.222247:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829477.222251:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4026 00000100:00080000:2.0:1787829477.222255:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829477.222259:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829477.222263:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829477.222265:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829477/real 1787829477] req@ffff8ab3f043b840 x1874671280891520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829477.222276:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280891904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829477.222289:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829477.222291:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280891520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829477.222295:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829477.222310:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829477.222320:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4026 00000100:00080000:1.0:1787829477.222325:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829477.222327:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829477.222335:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787829477.222336:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829477.222344:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280891520:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787829477.222346:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829477.222363:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280892032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829477.222373:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280891648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829477.222400:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829477.222414:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829477/real 1787829477] req@ffff8ab3f043b480 x1874671280891648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829477.222428:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280891648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829477.222437:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829477.222441:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829477.222444:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280891648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829477.222450:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280891776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829477.222470:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829477.222482:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829477/real 1787829477] req@ffff8ab3f0439e00 x1874671280891776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829477.222493:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280891776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829477.222509:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829477.222513:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829477.222516:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280891776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829477.222523:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280891904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829477.222545:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829477.222558:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829477/real 1787829477] req@ffff8ab3f043a1c0 x1874671280891904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829477.222569:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280891904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829477.222576:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829477.222578:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829477.222580:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280891904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829477.222593:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280892032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829477.222607:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829477.222614:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829477/real 1787829477] req@ffff8ab3f043a940 x1874671280892032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829477.222621:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280892032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829477.222628:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829477.222630:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829477.222632:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280892032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829478.246115:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829479.270067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829480.294075:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829481.318205:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829482.342037:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829482.342047:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829482.342052:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829482.342062:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4031 00010000:00010000:2.0:1787829482.342073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829482.342074:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829482.342084:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829482.342122:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280892160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829482.342149:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829482.342152:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829482.342155:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829482.342159:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4031 00000100:00080000:3.0:1787829482.342164:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829482.342170:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829482.342179:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280892160:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787829482.342180:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829482.342196:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280892288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829482.342208:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829482.342212:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829482.342214:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829482.342218:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4031 00000100:00080000:3.0:1787829482.342230:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829482.342235:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829482.342239:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829482.342254:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280892416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829482.342280:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829482.342283:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829482.342285:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829482.342290:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4031 00000100:00100000:1.0:1787829482.342295:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280892288:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787829482.342297:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829482.342306:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829482.342312:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829482.342327:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280892544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829482.342336:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280892416:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787829482.342354:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829482.342358:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829482.342361:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829482.342367:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4031 00000100:00080000:3.0:1787829482.342371:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829482.342372:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280892544:192.168.203.124@tcp:8:kworker.0 00000100:00080000:3.0:1787829482.342378:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829482.342385:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829482.342398:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280892672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829482.342443:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280892672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829483.366084:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829484.390076:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829485.414053:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787829485.414126:0:10942:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787829485.414161:0:10942:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829485.414179:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829485.414214:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829485.414236:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787829485.414254:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829482/real 1787829485] req@ffff8ab3d0a430c0 x1874671280892288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787829485.414267:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787829485.414283:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280892288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787829485.414286:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00080000:1.0:1787829485.414302:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829485.414310:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829485.414315:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280892288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829485.414324:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829482/real 1787829485] req@ffff8ab3d0a43c00 x1874671280892416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829485.414346:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280892416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829485.414357:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829485.414361:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829485.414365:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280892416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829485.414372:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829482/real 1787829485] req@ffff8ab3d0a41e00 x1874671280892160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829485.414385:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280892160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829485.414394:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829485.414397:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829485.414401:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280892160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829485.414425:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829482/real 1787829485] req@ffff8ab3d0a421c0 x1874671280892672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829485.414438:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280892672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829485.414449:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829485.414452:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829485.414455:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280892672:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829485.414461:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829482/real 1787829485] req@ffff8ab3d0a40000 x1874671280892544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829485.414472:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280892544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829485.414481:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829485.414484:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829485.414487:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280892544:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829486.438086:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829487.461994:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829487.462011:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829487.462015:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829487.462026:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4037 00000100:00080000:3.0:1787829487.462037:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1787829487.462041:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829487.462046:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829487.462082:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280892800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829487.462109:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829487.462113:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829487.462115:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829487.462119:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4037 00000100:00080000:3.0:1787829487.462123:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829487.462128:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829487.462140:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829487.462156:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280892800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829487.462157:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280892928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829487.462175:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829487.462178:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829487.462180:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829487.462185:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4037 00000100:00080000:3.0:1787829487.462189:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829487.462194:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829487.462199:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829487.462213:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280893056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829487.462224:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829487.462244:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829487.462245:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829487.462249:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829487.462256:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4037 00000100:00080000:3.0:1787829487.462261:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829487.462267:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829487.462273:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829487.462289:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280893184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829487.462299:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829487.462302:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829487.462305:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829487.462309:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4037 00000100:00080000:3.0:1787829487.462313:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829487.462318:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829487.462324:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829487/real 1787829487] req@ffff8ab3d0a412c0 x1874671280892800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829487.462326:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829487.462345:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280893312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829487.462352:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280892800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829487.462370:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829487.462376:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829487.462382:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280892800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829487.462409:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280892928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829487.462436:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829487.462453:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829487/real 1787829487] req@ffff8ab3d0a43840 x1874671280892928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829487.462465:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280892928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829487.462474:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829487.462477:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829487.462481:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280892928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829487.462487:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280893056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829487.462509:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829487.462520:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829487/real 1787829487] req@ffff8ab3d0a43480 x1874671280893056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829487.462530:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280893056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829487.462539:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829487.462543:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829487.462546:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280893056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829487.462553:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280893184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829487.462573:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829487.462585:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829487/real 1787829487] req@ffff8ab3d0a40000 x1874671280893184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829487.462598:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280893184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829487.462608:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829487.462611:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829487.462615:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280893184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829487.462622:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280893312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829487.462641:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829487.462652:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829487/real 1787829487] req@ffff8ab3d0a421c0 x1874671280893312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829487.462663:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280893312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829487.462678:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829487.462682:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829487.462685:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280893312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829488.486031:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829489.510061:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829490.534073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829491.558083:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829492.581997:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829492.582009:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829492.582013:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829492.582023:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4042 00000100:00080000:3.0:1787829492.582035:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829492.582042:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829492.582076:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280893440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787829492.582091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829492.582098:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829492.582101:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829492.582103:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829492.582106:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4042 00000100:00080000:3.0:1787829492.582111:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829492.582118:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829492.582127:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829492.582137:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280893440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829492.582139:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280893568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829492.582153:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829492.582155:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829492.582156:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829492.582159:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4042 00000100:00080000:3.0:1787829492.582161:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829492.582164:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829492.582168:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829492.582175:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280893696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829492.582182:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829492.582184:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829492.582185:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829492.582188:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4042 00000100:00080000:3.0:1787829492.582210:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829492.582214:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829492.582218:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829492.582222:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829492.582239:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280893824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829492.582247:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829492.582249:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829492.582251:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829492.582255:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4042 00000100:00080000:3.0:1787829492.582258:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829492.582262:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829492.582265:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829492.582267:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829492/real 1787829492] req@ffff8ab3d0a41e00 x1874671280893440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829492.582276:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280893952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829492.582289:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280893440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829492.582299:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829492.582304:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829492.582309:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280893440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829492.582334:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280893568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829492.582358:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829492.582373:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829492/real 1787829492] req@ffff8ab3d0a43c00 x1874671280893568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829492.582387:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280893568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829492.582396:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829492.582398:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829492.582400:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280893568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829492.582405:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280893696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829492.582421:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829492.582428:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829492/real 1787829492] req@ffff8ab3d0a430c0 x1874671280893696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829492.582436:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280893696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829492.582441:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829492.582452:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829492.582454:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280893696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829492.582458:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280893824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829492.582470:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829492.582476:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829492/real 1787829492] req@ffff8ab3d0a421c0 x1874671280893824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829492.582483:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280893824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829492.582489:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829492.582491:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829492.582493:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280893824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829492.582496:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280893952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829492.582505:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829492.582511:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829492/real 1787829492] req@ffff8ab3d0a40000 x1874671280893952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829492.582518:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280893952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829492.582524:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829492.582526:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829492.582528:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280893952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829493.606064:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829494.630083:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829495.654035:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829496.678091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829497.701991:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829497.702000:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829497.702008:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829497.702016:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4047 00000100:00080000:0.0:1787829497.702025:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829497.702031:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829497.702039:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829497.702063:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280894080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829497.702084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829497.702089:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829497.702090:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829497.702093:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4047 00000100:00080000:0.0:1787829497.702096:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829497.702099:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829497.702107:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829497.702120:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280894080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829497.702121:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280894208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829497.702136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829497.702138:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829497.702139:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829497.702142:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4047 00000100:00080000:0.0:1787829497.702144:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829497.702146:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829497.702150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829497.702174:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280894336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829497.702181:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829497.702183:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829497.702184:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829497.702187:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4047 00000100:00080000:0.0:1787829497.702191:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829497.702193:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829497.702198:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829497.702202:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829497.702210:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280894464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829497.702219:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829497.702221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829497.702223:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829497.702226:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4047 00000100:00080000:0.0:1787829497.702228:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829497.702230:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829497.702235:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829497.702242:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280894592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829497.702255:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829497/real 1787829497] req@ffff8ab3f075f480 x1874671280894080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829497.702271:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280894080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829497.702280:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829497.702283:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829497.702286:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280894080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829497.702304:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280894208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829497.702319:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829497.702328:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829497/real 1787829497] req@ffff8ab3f075f840 x1874671280894208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829497.702336:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280894208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829497.702346:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829497.702348:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829497.702351:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280894208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829497.702354:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280894336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829497.702364:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829497.702370:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829497/real 1787829497] req@ffff8ab3f075ed00 x1874671280894336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829497.702377:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280894336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829497.702382:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829497.702383:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829497.702385:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280894336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829497.702389:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280894464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829497.702398:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829497.702403:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829497/real 1787829497] req@ffff8ab3f075e580 x1874671280894464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829497.702410:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280894464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829497.702416:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829497.702418:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829497.702420:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280894464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829497.702424:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280894592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829497.702433:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829497.702438:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829497/real 1787829497] req@ffff8ab3f075e940 x1874671280894592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829497.702444:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280894592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829497.702450:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829497.702452:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829497.702454:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280894592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829498.726090:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829499.750107:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829500.774069:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829501.798067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829502.821995:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829502.822008:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829502.822013:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829502.822024:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4052 00000100:00080000:3.0:1787829502.822035:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829502.822044:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829502.822083:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829502.822090:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280894720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829502.822112:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829502.822116:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829502.822127:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829502.822131:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4052 00000100:00080000:3.0:1787829502.822135:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829502.822139:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829502.822150:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829502.822160:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280894720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829502.822162:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280894848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829502.822171:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829502.822175:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829502.822176:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829502.822180:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4052 00000100:00080000:3.0:1787829502.822183:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829502.822186:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829502.822190:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829502.822198:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280894976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829502.822207:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829502.822210:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829502.822211:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829502.822215:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4052 00000100:00080000:3.0:1787829502.822224:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829502.822228:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829502.822233:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829502.822234:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829502.822244:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280895104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829502.822252:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829502.822255:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829502.822257:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829502.822261:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4052 00000100:00080000:3.0:1787829502.822263:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829502.822267:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829502.822278:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829502.822287:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280895232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829502.822291:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829502/real 1787829502] req@ffff8ab3d0a43480 x1874671280894720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829502.822308:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280894720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829502.822320:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829502.822324:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829502.822329:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280894720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829502.822351:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280894848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829502.822368:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829502.822386:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829502/real 1787829502] req@ffff8ab3d0a43840 x1874671280894848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829502.822393:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280894848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829502.822399:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829502.822401:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829502.822403:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280894848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829502.822406:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280894976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829502.822417:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829502.822422:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829502/real 1787829502] req@ffff8ab3d0a412c0 x1874671280894976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829502.822429:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280894976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829502.822433:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829502.822435:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829502.822437:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280894976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829502.822440:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280895104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829502.822450:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829502.822455:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829502/real 1787829502] req@ffff8ab3d0a40000 x1874671280895104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829502.822461:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280895104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829502.822473:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829502.822475:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829502.822477:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280895104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829502.822481:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280895232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829502.822491:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829502.822496:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829502/real 1787829502] req@ffff8ab3d0a421c0 x1874671280895232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829502.822502:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280895232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829502.822508:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829502.822510:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829502.822512:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280895232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829503.846093:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829504.870085:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829505.894039:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829506.918039:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829507.941985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829507.941994:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829507.941997:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829507.942005:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4057 00000100:00080000:1.0:1787829507.942014:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829507.942020:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829507.942051:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280895360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829507.942073:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829507.942076:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829507.942078:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829507.942080:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4057 00000100:00080000:1.0:1787829507.942095:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829507.942098:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829507.942107:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829507.942118:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280895488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829507.942126:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280895360:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787829507.942129:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829507.942135:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829507.942136:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829507.942140:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4057 00000100:00080000:1.0:1787829507.942143:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829507.942145:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829507.942149:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829507.942157:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280895616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829507.942164:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829507.942167:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829507.942168:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829507.942171:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4057 00000100:00080000:1.0:1787829507.942179:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829507.942182:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829507.942186:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787829507.942188:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829507.942198:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280895744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829507.942206:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829507.942209:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829507.942210:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829507.942212:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4057 00000100:00080000:1.0:1787829507.942214:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829507.942216:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829507.942219:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829507.942229:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280895872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829507.942252:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829507/real 1787829507] req@ffff8ab3d22fd680 x1874671280895360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829507.942275:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280895360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829507.942287:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829507.942293:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829507.942298:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280895360:192.168.203.124@tcp:250:kworker.0 00010000:00010000:0.0:1787829507.942300:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829507.942321:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280895488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829507.942343:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829507.942354:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829507/real 1787829507] req@ffff8ab3d22fcb40 x1874671280895488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829507.942363:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280895488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829507.942369:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829507.942371:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829507.942373:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280895488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829507.942377:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280895616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829507.942387:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829507.942393:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829507/real 1787829507] req@ffff8ab3d22fda40 x1874671280895616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829507.942400:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280895616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829507.942405:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829507.942407:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829507.942409:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280895616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829507.942413:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280895744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829507.942432:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829507.942438:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829507/real 1787829507] req@ffff8ab3d22fc000 x1874671280895744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829507.942446:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280895744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829507.942453:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829507.942455:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829507.942457:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280895744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829507.942461:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280895872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829507.942472:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829507.942477:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829507/real 1787829507] req@ffff8ab3d22ff480 x1874671280895872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829507.942484:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280895872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829507.942490:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829507.942492:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829507.942494:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280895872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829508.966051:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829509.990086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829511.014102:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829512.038030:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829513.061994:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829513.062005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829513.062009:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829513.062019:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4062 00000100:00080000:1.0:1787829513.062030:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829513.062037:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829513.062099:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280896000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829513.062133:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829513.062138:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829513.062139:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829513.062142:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4062 00000100:00080000:1.0:1787829513.062145:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829513.062149:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829513.062159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829513.062171:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280896128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829513.062179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829513.062182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829513.062183:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829513.062187:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4062 00000100:00080000:1.0:1787829513.062189:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829513.062192:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829513.062195:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829513.062205:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280896256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829513.062206:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280896000:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787829513.062243:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829513.062248:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829513.062250:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829513.062256:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4062 00000100:00080000:1.0:1787829513.062261:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829513.062266:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829513.062272:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829513.062284:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280896384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829513.062300:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:1.0:1787829513.062307:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829513.062311:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829513.062313:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829513.062317:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4062 00000100:00080000:1.0:1787829513.062321:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829513.062324:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829513.062328:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829513.062337:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280896512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829513.062387:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829513/real 1787829513] req@ffff8ab3d22ff480 x1874671280896000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829513.062416:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280896000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829513.062428:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829513.062431:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829513.062435:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280896000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829513.062446:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280896128:192.168.203.124@tcp:38:kworker.0 00010000:00010000:3.0:1787829513.062453:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1787829513.062474:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829513.062498:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829513/real 1787829513] req@ffff8ab3d22fc000 x1874671280896128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829513.062513:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280896128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829513.062523:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829513.062526:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829513.062530:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280896128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829513.062558:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280896256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829513.062579:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829513.062591:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829513/real 1787829513] req@ffff8ab3d22fda40 x1874671280896256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829513.062617:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280896256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829513.062634:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829513.062637:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829513.062639:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280896256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829513.062644:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280896384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829513.062669:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829513.062677:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829513/real 1787829513] req@ffff8ab3d22fcb40 x1874671280896384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829513.062685:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280896384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829513.062692:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829513.062695:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829513.062697:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280896384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829513.062701:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280896512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829513.062711:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829513.062717:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829513/real 1787829513] req@ffff8ab3d22fd680 x1874671280896512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829513.062725:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280896512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829513.062735:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829513.062738:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829513.062742:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280896512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829514.086128:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829515.110057:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829516.134095:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829517.158098:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829518.182034:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829518.182042:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829518.182046:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829518.182055:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4067 00000100:00080000:3.0:1787829518.182063:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829518.182070:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829518.182082:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829518.182105:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280896640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829518.182127:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829518.182129:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829518.182131:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829518.182133:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4067 00000100:00080000:3.0:1787829518.182136:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829518.182138:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829518.182145:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829518.182153:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280896768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829518.182161:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829518.182163:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829518.182166:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829518.182169:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4067 00000100:00080000:3.0:1787829518.182171:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829518.182173:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829518.182176:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829518.182185:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280896896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829518.182187:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280896640:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829518.182197:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829518.182201:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829518.182203:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829518.182207:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4067 00000100:00080000:3.0:1787829518.182210:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829518.182215:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829518.182220:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829518.182236:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280897024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829518.182247:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829518.182251:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829518.182253:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829518.182259:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4067 00000800:00000100:2.0:1787829518.182261:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829518.182265:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829518.182271:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829518.182277:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829518.182292:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280897152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829518.182327:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829518/real 1787829518] req@ffff8ab3d0a430c0 x1874671280896640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829518.182348:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280896640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829518.182360:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829518.182364:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829518.182368:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280896640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829518.182376:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280896768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829518.182400:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829518.182423:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829518/real 1787829518] req@ffff8ab3d0a43c00 x1874671280896768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829518.182437:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280896768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829518.182446:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829518.182450:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829518.182454:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280896768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829518.182477:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280896896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829518.182510:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829518.182524:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829518/real 1787829518] req@ffff8ab3d0a41e00 x1874671280896896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829518.182537:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280896896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829518.182546:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829518.182549:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829518.182553:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280896896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829518.182559:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280897024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829518.182579:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829518.182591:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829518/real 1787829518] req@ffff8ab3d0a421c0 x1874671280897024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829518.182604:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280897024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829518.182614:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829518.182617:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829518.182621:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280897024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829518.182627:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280897152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829518.182645:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829518.182656:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829518/real 1787829518] req@ffff8ab3d0a40000 x1874671280897152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829518.182677:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280897152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829518.182687:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829518.182691:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829518.182694:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280897152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829519.206097:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829520.230106:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829521.254105:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829522.278037:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829523.302025:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829523.302036:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829523.302040:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829523.302049:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4072 00000100:00080000:3.0:1787829523.302058:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829523.302064:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829523.302071:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829523.302102:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280897280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829523.302122:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829523.302125:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829523.302126:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829523.302130:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4072 00000100:00080000:3.0:1787829523.302132:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829523.302135:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829523.302143:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829523.302149:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280897280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829523.302154:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280897408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829523.302164:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829523.302166:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829523.302168:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829523.302170:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4072 00000100:00080000:3.0:1787829523.302173:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829523.302176:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829523.302180:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829523.302189:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280897536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829523.302196:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829523.302199:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829523.302200:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829523.302202:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4072 00000100:00080000:3.0:1787829523.302205:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829523.302209:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829523.302213:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829523.302218:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829523.302228:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280897664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829523.302234:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829523.302237:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829523.302239:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829523.302241:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4072 00000100:00080000:3.0:1787829523.302244:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829523.302247:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829523.302250:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829523.302254:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829523/real 1787829523] req@ffff8ab3d0a412c0 x1874671280897280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829523.302261:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280897792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829523.302285:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280897280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829523.302295:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829523.302298:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829523.302300:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280897280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829523.302318:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280897408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829523.302333:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829523.302341:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829523/real 1787829523] req@ffff8ab3d0a43840 x1874671280897408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829523.302355:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280897408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829523.302361:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829523.302363:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829523.302365:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280897408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829523.302369:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280897536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829523.302378:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829523.302384:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829523/real 1787829523] req@ffff8ab3d0a43480 x1874671280897536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829523.302391:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280897536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829523.302397:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829523.302399:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829523.302401:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280897536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829523.302404:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280897664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829523.302421:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829523.302431:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829523/real 1787829523] req@ffff8ab3d0a40000 x1874671280897664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829523.302443:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280897664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829523.302461:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829523.302464:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829523.302467:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280897664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829523.302472:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280897792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829523.302486:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829523.302494:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829523/real 1787829523] req@ffff8ab3d0a421c0 x1874671280897792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829523.302501:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280897792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829523.302508:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829523.302510:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829523.302512:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280897792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829524.326062:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829525.350069:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829526.374028:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829527.398153:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829528.421990:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829528.421999:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829528.422002:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829528.422019:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4078 00000100:00080000:3.0:1787829528.422028:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829528.422035:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829528.422057:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829528.422068:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280897920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829528.422089:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829528.422092:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829528.422093:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829528.422096:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4078 00000100:00080000:3.0:1787829528.422098:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829528.422101:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829528.422108:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829528.422117:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280898048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829528.422125:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829528.422127:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829528.422130:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829528.422131:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280897920:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829528.422135:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4078 00000100:00080000:3.0:1787829528.422141:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829528.422144:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829528.422147:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829528.422156:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280898176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829528.422164:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829528.422166:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829528.422168:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829528.422170:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4078 00000100:00080000:3.0:1787829528.422183:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829528.422187:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829528.422190:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787829528.422193:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829528.422203:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280898304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829528.422213:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829528.422216:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829528.422218:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829528.422223:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4078 00000100:00080000:3.0:1787829528.422228:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829528.422235:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829528/real 1787829528] req@ffff8ab3d0a41e00 x1874671280897920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829528.422235:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829528.422245:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829528.422255:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280897920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829528.422263:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280898432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829528.422266:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829528.422274:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829528.422291:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280897920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829528.422313:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280898048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829528.422328:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829528.422338:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829528/real 1787829528] req@ffff8ab3d0a43c00 x1874671280898048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829528.422346:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280898048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829528.422352:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829528.422355:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829528.422357:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280898048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829528.422361:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280898176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829528.422380:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829528.422387:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829528/real 1787829528] req@ffff8ab3d0a430c0 x1874671280898176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829528.422394:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280898176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829528.422399:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829528.422401:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829528.422403:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280898176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829528.422407:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280898304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829528.422418:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829528.422423:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829528/real 1787829528] req@ffff8ab3d0a421c0 x1874671280898304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829528.422431:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280898304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829528.422437:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829528.422439:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829528.422440:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280898304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829528.422444:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280898432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829528.422453:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829528.422459:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829528/real 1787829528] req@ffff8ab3d0a40000 x1874671280898432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829528.422466:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280898432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829528.422472:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829528.422473:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829528.422475:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280898432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829529.446134:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829530.470048:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829531.494124:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829532.518092:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829533.542014:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829533.542025:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829533.542029:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829533.542040:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4083 00000100:00080000:2.0:1787829533.542050:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829533.542057:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829533.542079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829533.542099:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280898560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829533.542121:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829533.542123:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829533.542125:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829533.542127:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4083 00000100:00080000:2.0:1787829533.542129:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829533.542132:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829533.542140:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829533.542149:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280898688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829533.542157:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829533.542159:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829533.542161:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829533.542165:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280898560:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829533.542166:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4083 00000100:00080000:2.0:1787829533.542177:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829533.542182:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829533.542187:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829533.542201:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280898816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829533.542216:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829533.542220:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829533.542222:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829533.542227:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4083 00000100:00080000:2.0:1787829533.542233:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829533.542238:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829533.542244:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829533.542247:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829533.542257:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280898944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829533.542264:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829533.542267:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829533.542268:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829533.542271:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4083 00000100:00080000:2.0:1787829533.542275:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829533.542278:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829533.542281:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829533.542288:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280899072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829533.542312:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829533/real 1787829533] req@ffff8ab3f04392c0 x1874671280898560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829533.542333:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280898560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829533.542346:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829533.542350:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829533.542355:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280898560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829533.542381:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280898688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829533.542405:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829533.542418:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829533/real 1787829533] req@ffff8ab3f04383c0 x1874671280898688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829533.542432:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280898688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829533.542441:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829533.542444:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829533.542448:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280898688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829533.542455:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280898816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829533.542474:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829533.542485:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829533/real 1787829533] req@ffff8ab3f043ad00 x1874671280898816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829533.542497:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280898816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829533.542506:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829533.542510:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829533.542513:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280898816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829533.542519:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280898944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829533.542539:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829533.542550:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829533/real 1787829533] req@ffff8ab3f043bc00 x1874671280898944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829533.542562:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280898944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829533.542579:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829533.542583:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829533.542587:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280898944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829533.542593:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280899072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829533.542611:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829533.542622:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829533/real 1787829533] req@ffff8ab3f0439a40 x1874671280899072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829533.542633:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280899072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829533.542643:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829533.542647:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829533.542650:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280899072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829534.566031:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829535.590100:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829536.614017:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829537.638033:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829538.662012:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829538.662027:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829538.662033:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829538.662045:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4088 00000100:00080000:3.0:1787829538.662059:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1787829538.662066:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829538.662069:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829538.662114:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280899200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829538.662136:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829538.662139:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829538.662140:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829538.662143:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4088 00000100:00080000:3.0:1787829538.662146:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829538.662148:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829538.662157:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829538.662166:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280899328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829538.662173:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829538.662177:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829538.662179:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280899200:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829538.662179:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829538.662183:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4088 00000100:00080000:3.0:1787829538.662188:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829538.662190:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829538.662193:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829538.662201:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280899456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829538.662213:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829538.662217:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829538.662219:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829538.662223:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4088 00000100:00080000:3.0:1787829538.662244:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829538.662249:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829538.662255:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829538.662256:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829538.662271:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280899584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829538.662285:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829538.662290:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829538.662292:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829538.662296:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4088 00000100:00080000:3.0:1787829538.662300:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829538.662305:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829538.662310:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829538.662324:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280899712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829538.662336:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829538/real 1787829538] req@ffff8ab3d0a43480 x1874671280899200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829538.662357:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280899200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829538.662369:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829538.662374:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829538.662386:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280899200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829538.662413:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280899328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829538.662435:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829538.662454:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829538/real 1787829538] req@ffff8ab3d0a43840 x1874671280899328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829538.662467:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280899328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829538.662475:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829538.662479:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829538.662482:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280899328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829538.662488:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280899456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829538.662508:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829538.662519:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829538/real 1787829538] req@ffff8ab3d0a412c0 x1874671280899456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829538.662531:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280899456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829538.662540:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829538.662543:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829538.662546:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280899456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829538.662552:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280899584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829538.662572:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829538.662583:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829538/real 1787829538] req@ffff8ab3d0a40000 x1874671280899584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829538.662595:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280899584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829538.662605:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829538.662608:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829538.662611:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280899584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829538.662616:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280899712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829538.662630:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829538.662637:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829538/real 1787829538] req@ffff8ab3d0a421c0 x1874671280899712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829538.662644:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280899712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829538.662651:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829538.662654:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829538.662657:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280899712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829539.686059:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829540.710063:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829541.734043:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829542.758071:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829543.782014:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829543.782023:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829543.782027:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829543.782036:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4093 00000100:00080000:2.0:1787829543.782045:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1787829543.782045:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829543.782054:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829543.782088:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280899840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829543.782107:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829543.782110:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829543.782111:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829543.782114:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4093 00000100:00080000:2.0:1787829543.782117:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829543.782120:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829543.782128:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829543.782137:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280899840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829543.782139:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280899968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829543.782151:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829543.782153:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829543.782155:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829543.782158:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4093 00000100:00080000:2.0:1787829543.782160:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829543.782164:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829543.782169:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829543.782182:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280900096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829543.782192:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829543.782196:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829543.782198:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829543.782202:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829543.782206:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4093 00000100:00080000:2.0:1787829543.782213:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829543.782219:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829543.782225:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829543.782238:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280900224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829543.782249:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829543.782250:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829543/real 1787829543] req@ffff8ab3f0439680 x1874671280899840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829543.782260:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829543.782266:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829543.782273:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4093 00000100:00100000:1.0:1787829543.782274:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280899840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829543.782278:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829543.782288:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829543.782288:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787829543.782296:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000008:00100000:2.0:1787829543.782298:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829543.782299:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280899840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829543.782318:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280900352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829543.782324:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280899968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829543.782347:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829543.782358:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829543/real 1787829543] req@ffff8ab3f043b0c0 x1874671280899968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829543.782366:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280899968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829543.782372:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829543.782374:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829543.782377:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280899968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829543.782380:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280900096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829543.782390:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829543.782395:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829543/real 1787829543] req@ffff8ab3f043a580 x1874671280900096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829543.782402:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280900096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829543.782407:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829543.782409:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829543.782412:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280900096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829543.782416:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280900224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829543.782426:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829543.782431:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829543/real 1787829543] req@ffff8ab3f0438f00 x1874671280900224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829543.782438:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280900224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829543.782444:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829543.782446:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829543.782447:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280900224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829543.782457:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280900352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829543.782466:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829543.782472:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829543/real 1787829543] req@ffff8ab3f0438b40 x1874671280900352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829543.782479:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280900352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829543.782491:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829543.782493:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829543.782495:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280900352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829544.806070:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829545.830074:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829546.854059:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829547.878061:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829548.902013:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829548.902021:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829548.902025:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829548.902034:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4098 00000100:00080000:3.0:1787829548.902043:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829548.902049:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829548.902071:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829548.902083:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280900480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829548.902109:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829548.902112:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829548.902114:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829548.902116:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4098 00000100:00080000:3.0:1787829548.902119:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829548.902122:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829548.902131:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829548.902143:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280900608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829548.902144:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280900480:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829548.902161:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829548.902165:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829548.902167:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829548.902172:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4098 00000100:00080000:3.0:1787829548.902176:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829548.902181:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829548.902187:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829548.902200:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280900736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829548.902212:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829548.902214:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829548.902216:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829548.902219:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4098 00000100:00080000:3.0:1787829548.902223:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829548.902227:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829548.902233:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829548.902245:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280900864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829548.902251:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280900608:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787829548.902255:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829548.902259:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829548.902260:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829548.902265:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4098 00000100:00080000:3.0:1787829548.902268:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829548.902271:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829548.902275:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829548.902281:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280900736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787829548.902287:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280900992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829548.902307:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280900864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829548.902323:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280900992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829549.926076:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829550.950066:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829551.974037:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787829551.974279:0:10944:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787829551.974333:0:10944:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829551.974350:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829551.974402:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829551.974412:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829551.974429:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829551.974440:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787829551.974480:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829548/real 1787829551] req@ffff8ab3d0a421c0 x1874671280900864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829551.974517:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280900864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829551.974539:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829551.974546:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829551.974551:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280900864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829551.974560:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829548/real 1787829551] req@ffff8ab3d0a40000 x1874671280900992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829551.974574:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280900992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829551.974589:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829551.974593:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829551.974596:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280900992:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829551.974604:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829548/real 1787829551] req@ffff8ab3d0a43c00 x1874671280900608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829551.974617:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280900608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829551.974626:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829551.974630:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829551.974633:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280900608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829551.974641:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829548/real 1787829551] req@ffff8ab3d0a41e00 x1874671280900736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829551.974662:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280900736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829551.974671:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829551.974675:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829551.974678:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280900736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829551.974685:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829548/real 1787829551] req@ffff8ab3d0a430c0 x1874671280900480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829551.974697:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280900480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829551.974705:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829551.974708:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829551.974807:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280900480:192.168.203.124@tcp:250:kworker.0 00010000:00010000:0.0:1787829552.998054:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829554.021973:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829554.021989:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829554.021995:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829554.022009:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4103 00000100:00080000:3.0:1787829554.022030:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787829554.022036:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829554.022040:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829554.022102:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280901120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829554.022137:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829554.022141:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829554.022143:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829554.022147:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4103 00000100:00080000:3.0:1787829554.022152:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829554.022156:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829554.022169:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829554.022187:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280901248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829554.022189:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280901120:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829554.022208:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829554.022211:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829554.022213:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829554.022218:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4103 00000100:00080000:3.0:1787829554.022223:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829554.022226:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829554.022232:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829554.022245:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280901376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829554.022260:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829554.022264:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829554.022266:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829554.022271:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4103 00000100:00080000:3.0:1787829554.022275:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829554.022282:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829554.022283:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829554.022292:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829554.022307:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280901504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829554.022318:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829554.022322:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829554.022324:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829554.022329:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4103 00000100:00080000:3.0:1787829554.022334:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829554.022340:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829554.022346:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829554.022349:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829554/real 1787829554] req@ffff8ab3d0a412c0 x1874671280901120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829554.022361:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280901632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829554.022381:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280901120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829554.022396:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829554.022402:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829554.022407:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280901120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829554.022451:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280901248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829554.022479:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829554.022495:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829554/real 1787829554] req@ffff8ab3d0a43840 x1874671280901248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829554.022509:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280901248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829554.022519:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829554.022523:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829554.022529:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280901248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829554.022535:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280901376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829554.022556:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829554.022567:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829554/real 1787829554] req@ffff8ab3d0a43480 x1874671280901376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829554.022579:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280901376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829554.022594:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829554.022599:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829554.022602:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280901376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829554.022608:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280901504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829554.022628:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829554.022640:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829554/real 1787829554] req@ffff8ab3d0a430c0 x1874671280901504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829554.022651:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280901504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829554.022660:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829554.022663:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829554.022666:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280901504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829554.022672:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280901632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829554.022693:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829554.022704:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829554/real 1787829554] req@ffff8ab3d0a41e00 x1874671280901632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829554.022715:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280901632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829554.022725:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829554.022728:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829554.022731:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280901632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829555.046114:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829556.070133:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829557.094068:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829558.118111:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829559.141994:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829559.142009:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829559.142014:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829559.142027:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4108 00000100:00080000:3.0:1787829559.142042:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829559.142053:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829559.142070:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829559.142100:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280901760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829559.142124:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829559.142126:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829559.142128:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829559.142131:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4108 00000100:00080000:3.0:1787829559.142134:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829559.142138:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829559.142148:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829559.142161:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280901760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829559.142161:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280901888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829559.142176:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829559.142179:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829559.142180:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829559.142190:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4108 00000100:00080000:3.0:1787829559.142192:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829559.142195:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829559.142198:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829559.142208:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280902016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829559.142215:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829559.142218:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829559.142219:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829559.142223:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4108 00000100:00080000:3.0:1787829559.142242:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829559.142249:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829559.142254:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829559.142258:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829559.142272:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280902144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829559.142285:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829559.142289:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829559.142291:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829559.142295:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4108 00000100:00080000:3.0:1787829559.142298:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829559.142302:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829559.142306:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829559.142318:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280902272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829559.142331:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829559/real 1787829559] req@ffff8ab3d0a43c00 x1874671280901760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829559.142348:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280901760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829559.142362:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829559.142366:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829559.142369:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280901760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829559.142394:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280901888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829559.142412:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829559.142428:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829559/real 1787829559] req@ffff8ab3d0a40000 x1874671280901888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829559.142437:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280901888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829559.142443:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829559.142445:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829559.142448:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280901888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829559.142452:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280902016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829559.142463:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829559.142471:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829559/real 1787829559] req@ffff8ab3d0a421c0 x1874671280902016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829559.142479:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280902016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829559.142484:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829559.142486:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829559.142488:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280902016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829559.142492:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280902144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829559.142503:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829559.142509:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829559/real 1787829559] req@ffff8ab3d0a41e00 x1874671280902144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829559.142516:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280902144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829559.142529:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829559.142531:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829559.142533:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280902144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829559.142537:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280902272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829559.142548:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829559.142554:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829559/real 1787829559] req@ffff8ab3d0a430c0 x1874671280902272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829559.142562:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280902272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829559.142574:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829559.142576:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829559.142579:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280902272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829560.166101:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829561.190063:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829562.214094:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829563.238086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829564.262009:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829564.262024:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829564.262029:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829564.262041:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4113 00000100:00080000:3.0:1787829564.262052:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829564.262060:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829564.262108:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280902400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787829564.262112:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829564.262148:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829564.262153:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829564.262155:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829564.262161:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4113 00000100:00080000:3.0:1787829564.262166:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829564.262172:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829564.262186:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829564.262198:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280902400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787829564.262202:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280902528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829564.262222:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829564.262226:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829564.262229:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829564.262236:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4113 00000100:00080000:3.0:1787829564.262241:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829564.262247:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829564.262253:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829564.262272:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280902656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829564.262285:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829564.262291:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787829564.262309:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829564.262313:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829564.262321:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4113 00000100:00080000:3.0:1787829564.262328:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829564.262333:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829564.262340:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829564.262346:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829564/real 1787829564] req@ffff8ab3d0a43480 x1874671280902400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829564.262357:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280902784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829564.262364:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280902400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829564.262378:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829564.262383:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829564.262385:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829564.262388:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787829564.262391:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829564.262396:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280902400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829564.262405:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4113 00000100:00080000:3.0:1787829564.262412:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829564.262417:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829564.262423:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829564.262430:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280902528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787829564.262445:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280902912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829564.262463:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829564.262478:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829564/real 1787829564] req@ffff8ab3d0a43840 x1874671280902528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829564.262493:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280902528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829564.262503:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829564.262507:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829564.262511:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280902528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829564.262526:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280902656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829564.262546:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829564.262555:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829564/real 1787829564] req@ffff8ab3d0a412c0 x1874671280902656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829564.262563:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280902656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829564.262569:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829564.262572:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829564.262574:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280902656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829564.262578:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280902784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829564.262589:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829564.262595:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829564/real 1787829564] req@ffff8ab3d0a430c0 x1874671280902784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829564.262602:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280902784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829564.262608:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829564.262610:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829564.262612:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280902784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829564.262624:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280902912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829564.262635:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829564.262640:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829564/real 1787829564] req@ffff8ab3d0a41e00 x1874671280902912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829564.262648:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280902912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829564.262655:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829564.262656:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829564.262658:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280902912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829565.286051:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829566.310067:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829567.334053:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829568.358107:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829569.381979:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829569.381988:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829569.381992:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829569.382000:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4119 00000100:00080000:2.0:1787829569.382008:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829569.382015:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829569.382046:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829569.382048:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280903040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829569.382071:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829569.382073:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829569.382074:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829569.382077:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4119 00000100:00080000:2.0:1787829569.382080:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829569.382083:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829569.382091:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829569.382101:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280903040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829569.382104:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280903168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829569.382112:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829569.382117:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829569.382118:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829569.382121:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4119 00000100:00080000:2.0:1787829569.382123:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829569.382126:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829569.382129:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829569.382137:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280903296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829569.382144:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829569.382147:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829569.382148:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829569.382151:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4119 00000100:00080000:2.0:1787829569.382166:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829569.382172:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829569.382177:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829569.382180:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829569.382196:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280903424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829569.382207:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829569.382211:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829569.382213:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829569.382218:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4119 00000100:00080000:2.0:1787829569.382223:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829569.382231:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829569/real 1787829569] req@ffff8ab3f0438780 x1874671280903040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829569.382231:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829569.382237:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829569.382253:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280903552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829569.382256:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280903040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829569.382277:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829569.382281:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829569.382284:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280903040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829569.382303:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280903168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829569.382321:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829569.382330:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829569/real 1787829569] req@ffff8ab3f0438b40 x1874671280903168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829569.382343:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280903168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829569.382358:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829569.382362:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829569.382365:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280903168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829569.382372:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280903296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829569.382391:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829569.382401:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829569/real 1787829569] req@ffff8ab3f0438f00 x1874671280903296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829569.382414:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280903296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829569.382422:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829569.382426:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829569.382430:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280903296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829569.382436:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280903424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829569.382455:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829569.382463:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829569/real 1787829569] req@ffff8ab3f043a580 x1874671280903424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829569.382471:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280903424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829569.382478:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829569.382480:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829569.382484:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280903424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829569.382490:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280903552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829569.382508:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829569.382520:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829569/real 1787829569] req@ffff8ab3f043b0c0 x1874671280903552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829569.382532:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280903552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829569.382540:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829569.382542:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829569.382544:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280903552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829570.406119:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829571.430028:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829572.454081:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829573.477995:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829574.502003:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829574.502011:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829574.502016:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829574.502025:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4124 00000100:00080000:2.0:1787829574.502033:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829574.502039:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829574.502070:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280903680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787829574.502084:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829574.502091:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829574.502095:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829574.502103:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829574.502106:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4124 00000100:00080000:2.0:1787829574.502111:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829574.502115:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829574.502122:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829574.502134:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280903808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829574.502140:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280903680:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829574.502143:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829574.502150:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829574.502152:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829574.502156:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4124 00000100:00080000:2.0:1787829574.502158:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829574.502160:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829574.502163:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829574.502172:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280903936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829574.502178:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829574.502181:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829574.502182:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829574.502185:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4124 00000100:00080000:2.0:1787829574.502193:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829574.502196:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829574.502199:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829574.502204:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829574.502210:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280904064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829574.502219:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829574.502221:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829574.502222:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829574.502225:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4124 00000100:00080000:2.0:1787829574.502227:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829574.502230:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829574.502233:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829574.502242:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280904192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829574.502247:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829574/real 1787829574] req@ffff8ab3f0439680 x1874671280903680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829574.502264:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280903680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829574.502272:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829574.502275:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829574.502278:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280903680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829574.502296:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280903808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829574.502311:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829574.502320:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829574/real 1787829574] req@ffff8ab3f0439a40 x1874671280903808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829574.502328:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280903808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829574.502333:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829574.502335:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829574.502338:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280903808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829574.502341:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280903936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829574.502351:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829574.502356:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829574/real 1787829574] req@ffff8ab3f043bc00 x1874671280903936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829574.502364:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280903936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829574.502369:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829574.502370:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829574.502372:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280903936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829574.502376:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280904064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829574.502386:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829574.502391:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829574/real 1787829574] req@ffff8ab3f043ad00 x1874671280904064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829574.502399:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280904064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829574.502406:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829574.502408:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829574.502409:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280904064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829574.502413:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280904192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829574.502422:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829574.502427:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829574/real 1787829574] req@ffff8ab3f04383c0 x1874671280904192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829574.502434:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280904192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829574.502440:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829574.502442:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829574.502444:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280904192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829575.526053:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829576.550050:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829577.574075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829578.598104:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829579.622006:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829579.622014:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829579.622018:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829579.622027:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4129 00000100:00080000:2.0:1787829579.622037:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829579.622042:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829579.622076:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280904320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829579.622095:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829579.622098:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829579.622100:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829579.622104:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4129 00000100:00080000:2.0:1787829579.622107:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829579.622111:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829579.622121:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:2.0:1787829579.622123:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829579.622140:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280904320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829579.622144:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280904448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829579.622159:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829579.622163:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829579.622166:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829579.622170:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4129 00000100:00080000:2.0:1787829579.622188:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829579.622192:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829579.622198:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829579.622198:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829579.622214:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280904576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829579.622222:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829579.622224:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829579.622225:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829579.622228:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4129 00000100:00080000:2.0:1787829579.622231:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829579.622234:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829579.622239:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829579.622248:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280904704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829579.622257:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829579/real 1787829579] req@ffff8ab3f04392c0 x1874671280904320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829579.622257:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829579.622264:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829579.622266:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829579.622273:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4129 00000100:00080000:2.0:1787829579.622278:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829579.622283:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829579.622287:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829579.622297:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280904320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829579.622297:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280904832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829579.622341:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829579.622347:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829579.622352:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280904320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829579.622378:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280904448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829579.622403:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829579.622419:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829579/real 1787829579] req@ffff8ab3f043a940 x1874671280904448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829579.622432:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280904448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829579.622440:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829579.622442:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829579.622445:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280904448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829579.622450:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280904576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829579.622465:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829579.622473:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829579/real 1787829579] req@ffff8ab3f043a1c0 x1874671280904576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829579.622480:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280904576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829579.622486:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829579.622488:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829579.622490:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280904576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829579.622494:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280904704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829579.622504:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829579.622511:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829579/real 1787829579] req@ffff8ab3f0439e00 x1874671280904704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829579.622523:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280904704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829579.622533:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829579.622536:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829579.622539:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280904704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829579.622546:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280904832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829579.622567:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829579.622579:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829579/real 1787829579] req@ffff8ab3f043b480 x1874671280904832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829579.622611:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280904832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829579.622621:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829579.622625:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829579.622628:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280904832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829580.646069:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829581.670090:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829582.694051:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829583.718116:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829584.742001:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829584.742009:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829584.742013:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829584.742023:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4134 00000100:00080000:2.0:1787829584.742033:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829584.742039:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829584.742074:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280904960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787829584.742083:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829584.742097:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829584.742101:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829584.742103:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829584.742106:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4134 00000100:00080000:2.0:1787829584.742109:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829584.742112:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829584.742120:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829584.742131:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280905088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829584.742139:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280904960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829584.742141:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829584.742147:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829584.742150:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829584.742154:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4134 00000100:00080000:2.0:1787829584.742156:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829584.742158:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829584.742162:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829584.742171:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280905216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829584.742178:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829584.742180:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829584.742182:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829584.742184:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4134 00000100:00080000:2.0:1787829584.742198:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829584.742202:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829584.742205:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829584.742210:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829584.742214:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280905344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829584.742222:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829584.742224:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829584.742226:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829584.742229:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4134 00000100:00080000:2.0:1787829584.742233:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829584.742236:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829584.742238:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829584.742245:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280905472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829584.742271:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829584/real 1787829584] req@ffff8ab3f043b840 x1874671280904960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829584.742291:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280904960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829584.742304:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829584.742309:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829584.742313:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280904960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829584.742336:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280905088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829584.742361:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829584.742376:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829584/real 1787829584] req@ffff8ab3f043b480 x1874671280905088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829584.742390:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280905088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829584.742400:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829584.742403:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829584.742407:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280905088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829584.742413:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280905216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829584.742434:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829584.742447:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829584/real 1787829584] req@ffff8ab3f0439e00 x1874671280905216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829584.742460:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280905216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829584.742469:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829584.742472:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829584.742476:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280905216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829584.742482:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280905344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829584.742502:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829584.742514:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829584/real 1787829584] req@ffff8ab3f043a1c0 x1874671280905344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829584.742526:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280905344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829584.742536:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829584.742540:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829584.742543:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280905344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829584.742564:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280905472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829584.742585:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829584.742597:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829584/real 1787829584] req@ffff8ab3f043a940 x1874671280905472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829584.742610:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280905472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829584.742621:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829584.742624:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829584.742628:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280905472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829585.766049:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829586.790040:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829587.814084:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829588.838048:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829589.862019:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829589.862034:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829589.862040:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829589.862052:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4139 00000100:00080000:3.0:1787829589.862066:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829589.862075:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829589.862077:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829589.862124:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280905600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829589.862154:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829589.862158:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829589.862161:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829589.862165:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4139 00000100:00080000:3.0:1787829589.862170:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829589.862175:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829589.862186:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280905600:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787829589.862187:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829589.862204:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280905728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829589.862216:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829589.862220:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829589.862222:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829589.862227:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4139 00000100:00080000:3.0:1787829589.862245:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829589.862249:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829589.862255:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787829589.862256:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829589.862274:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280905856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829589.862286:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829589.862291:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829589.862293:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829589.862298:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4139 00000100:00080000:3.0:1787829589.862302:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829589.862306:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829589/real 1787829589] req@ffff8ab3d0a421c0 x1874671280905600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829589.862310:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829589.862321:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829589.862331:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280905600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829589.862336:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280905984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829589.862345:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829589.862352:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:3.0:1787829589.862354:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829589.862357:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280905600:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829589.862361:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829589.862365:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829589.862371:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4139 00000100:00080000:3.0:1787829589.862376:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829589.862380:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829589.862385:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280905728:192.168.203.124@tcp:38:kworker.0 00000008:00100000:3.0:1787829589.862386:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829589.862404:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280906112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829589.862408:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829589.862422:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829589/real 1787829589] req@ffff8ab3d0a40000 x1874671280905728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829589.862433:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280905728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829589.862439:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829589.862442:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829589.862445:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280905728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829589.862449:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280905856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829589.862458:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829589.862464:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829589/real 1787829589] req@ffff8ab3d0a43c00 x1874671280905856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829589.862472:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280905856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829589.862483:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829589.862485:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829589.862486:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280905856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829589.862490:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280905984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829589.862509:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829589.862530:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829589/real 1787829589] req@ffff8ab3d0a41e00 x1874671280905984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829589.862538:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280905984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829589.862546:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829589.862548:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829589.862551:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280905984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829589.862565:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280906112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829589.862578:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829589.862585:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829589/real 1787829589] req@ffff8ab3d0a430c0 x1874671280906112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829589.862592:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280906112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829589.862600:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829589.862602:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829589.862604:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280906112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829590.886066:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829591.910100:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829592.934063:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829593.958100:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829594.982005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829594.982018:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829594.982023:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829594.982035:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4144 00000100:00080000:0.0:1787829594.982049:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1787829594.982053:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829594.982058:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829594.982109:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280906240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829594.982137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829594.982140:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829594.982141:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829594.982146:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4144 00000100:00080000:0.0:1787829594.982151:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829594.982155:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829594.982167:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829594.982175:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280906240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829594.982183:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280906368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829594.982194:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829594.982197:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829594.982199:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829594.982203:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4144 00000100:00080000:0.0:1787829594.982207:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829594.982211:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829594.982216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829594.982227:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280906496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829594.982236:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829594.982240:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829594.982241:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829594.982246:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829594.982247:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4144 00000100:00080000:0.0:1787829594.982253:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829594.982259:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829594.982265:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829594.982276:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280906624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829594.982286:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829594.982290:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829594.982292:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829594.982296:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4144 00000100:00100000:1.0:1787829594.982300:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829594/real 1787829594] req@ffff8ab3f075da40 x1874671280906240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829594.982301:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829594.982310:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829594.982316:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829594.982320:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280906240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829594.982331:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829594.982332:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280906752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829594.982335:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829594.982342:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280906240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829594.982369:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280906368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829594.982383:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829594.982392:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829594/real 1787829594] req@ffff8ab3f075cb40 x1874671280906368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829594.982399:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280906368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829594.982404:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829594.982406:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829594.982408:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280906368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829594.982412:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280906496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829594.982421:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829594.982427:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829594/real 1787829594] req@ffff8ab3f075f0c0 x1874671280906496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829594.982433:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280906496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829594.982438:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829594.982440:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829594.982442:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280906496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829594.982445:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280906624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829594.982454:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829594.982459:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829594/real 1787829594] req@ffff8ab3f075de00 x1874671280906624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829594.982466:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280906624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829594.982472:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829594.982474:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829594.982475:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280906624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829594.982494:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280906752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829594.982508:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829594.982514:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829594/real 1787829594] req@ffff8ab3f075cf00 x1874671280906752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829594.982521:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280906752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829594.982528:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829594.982531:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829594.982534:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280906752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829596.006098:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829597.030089:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829598.054091:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829599.078073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829600.102012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829600.102024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829600.102028:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829600.102039:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4149 00000100:00080000:0.0:1787829600.102052:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829600.102059:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829600.102108:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280906880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787829600.102127:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829600.102141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829600.102146:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829600.102149:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829600.102153:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4149 00000100:00080000:0.0:1787829600.102156:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829600.102161:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829600.102174:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829600.102190:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280907008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829600.102201:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829600.102207:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280906880:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829600.102208:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829600.102212:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829600.102225:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4149 00000100:00080000:0.0:1787829600.102230:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829600.102234:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829600.102239:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829600.102255:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280907136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829600.102265:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829600.102268:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829600.102270:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829600.102275:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4149 00000100:00080000:0.0:1787829600.102283:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829600.102288:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829600.102295:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787829600.102296:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829600.102314:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280907264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829600.102323:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829600.102327:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829600.102328:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829600.102333:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4149 00000100:00080000:0.0:1787829600.102338:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829600.102342:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829600.102347:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829600.102360:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280907392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829600.102364:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829600/real 1787829600] req@ffff8ab3f075c000 x1874671280906880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829600.102397:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280906880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829600.102409:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829600.102414:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829600.102419:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280906880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829600.102459:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280907008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829600.102486:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829600.102501:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829600/real 1787829600] req@ffff8ab3f075cf00 x1874671280907008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829600.102514:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280907008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829600.102524:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829600.102528:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829600.102531:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280907008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829600.102539:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280907136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829600.102560:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829600.102572:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829600/real 1787829600] req@ffff8ab3f075de00 x1874671280907136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829600.102585:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280907136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829600.102594:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829600.102598:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829600.102601:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280907136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829600.102609:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280907264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829600.102630:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829600.102641:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829600/real 1787829600] req@ffff8ab3f075f0c0 x1874671280907264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829600.102654:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280907264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829600.102665:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829600.102669:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829600.102682:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280907264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829600.102693:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280907392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829600.102714:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829600.102727:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829600/real 1787829600] req@ffff8ab3f075cb40 x1874671280907392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829600.102739:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280907392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829600.102750:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829600.102753:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829600.102756:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280907392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829601.126204:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829602.150093:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829603.174105:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829604.198136:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829605.221969:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829605.221978:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829605.221982:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829605.221993:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4154 00000100:00080000:0.0:1787829605.222005:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829605.222014:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829605.222027:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829605.222054:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280907520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829605.222095:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829605.222098:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829605.222100:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829605.222103:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4154 00000100:00080000:0.0:1787829605.222108:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829605.222112:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829605.222120:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829605.222126:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280907520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829605.222136:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280907648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829605.222145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829605.222149:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829605.222150:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829605.222154:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4154 00000100:00080000:0.0:1787829605.222157:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829605.222159:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829605.222162:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829605.222170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280907776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829605.222178:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829605.222180:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829605.222184:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829605.222184:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829605.222189:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4154 00000100:00080000:0.0:1787829605.222195:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829605.222198:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829605.222202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829605.222211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280907904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829605.222218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829605.222220:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829605.222222:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829605.222225:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4154 00000100:00080000:0.0:1787829605.222228:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829605.222231:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829605/real 1787829605] req@ffff8ab3f075da40 x1874671280907520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829605.222234:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829605.222241:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829605.222254:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280908032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829605.222256:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280907520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829605.222281:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829605.222284:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829605.222287:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280907520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829605.222307:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280907648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829605.222321:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829605.222329:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829605/real 1787829605] req@ffff8ab3f075e940 x1874671280907648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829605.222337:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280907648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829605.222347:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829605.222350:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829605.222352:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280907648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829605.222355:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280907776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829605.222364:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829605.222370:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829605/real 1787829605] req@ffff8ab3f075e580 x1874671280907776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829605.222377:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280907776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829605.222382:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829605.222383:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829605.222385:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280907776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829605.222389:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280907904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829605.222399:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829605.222404:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829605/real 1787829605] req@ffff8ab3f075ed00 x1874671280907904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829605.222429:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280907904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829605.222436:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829605.222438:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829605.222440:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280907904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829605.222444:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280908032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829605.222455:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829605.222462:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829605/real 1787829605] req@ffff8ab3f075f840 x1874671280908032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829605.222469:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280908032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829605.222475:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829605.222476:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829605.222478:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280908032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829606.246063:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829607.270044:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829608.294053:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829609.318034:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829610.341957:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829610.341965:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829610.341969:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829610.341979:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4159 00000100:00080000:0.0:1787829610.341991:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829610.341999:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829610.342013:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829610.342038:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280908160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829610.342070:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829610.342076:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829610.342077:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829610.342080:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4159 00000100:00080000:0.0:1787829610.342084:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829610.342090:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829610.342100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829610.342110:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280908160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829610.342120:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280908288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829610.342134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829610.342138:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829610.342141:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829610.342146:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4159 00000100:00080000:0.0:1787829610.342164:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829610.342168:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829610.342173:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829610.342174:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829610.342187:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280908416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829610.342195:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829610.342198:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829610.342200:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829610.342203:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4159 00000100:00080000:0.0:1787829610.342206:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829610.342209:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829610.342213:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829610.342219:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829610/real 1787829610] req@ffff8ab3f075f480 x1874671280908160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829610.342222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280908544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829610.342249:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829610.342251:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829610.342253:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829610.342257:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280908160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829610.342259:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4159 00000100:00080000:0.0:1787829610.342264:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829610.342270:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829610.342276:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787829610.342277:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829610.342282:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787829610.342286:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280908672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829610.342290:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280908160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829610.342322:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280908288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829610.342347:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829610.342360:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829610/real 1787829610] req@ffff8ab3f075f840 x1874671280908288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829610.342369:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280908288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829610.342389:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829610.342391:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829610.342394:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280908288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829610.342399:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280908416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829610.342414:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829610.342422:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829610/real 1787829610] req@ffff8ab3f075ed00 x1874671280908416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829610.342429:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280908416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829610.342435:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829610.342437:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829610.342439:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280908416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829610.342443:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280908544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829610.342464:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829610.342471:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829610/real 1787829610] req@ffff8ab3f075e580 x1874671280908544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829610.342479:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280908544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829610.342486:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829610.342488:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829610.342490:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280908544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829610.342493:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280908672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829610.342503:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829610.342509:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829610/real 1787829610] req@ffff8ab3f075e940 x1874671280908672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829610.342515:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280908672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829610.342522:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829610.342523:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829610.342525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280908672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829611.366121:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829612.390033:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829613.414095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829614.438094:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829615.462011:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829615.462022:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829615.462027:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829615.462039:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4165 00000100:00080000:2.0:1787829615.462049:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829615.462058:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829615.462063:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829615.462098:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280908800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829615.462126:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829615.462130:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829615.462133:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829615.462139:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4165 00000100:00080000:2.0:1787829615.462144:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829615.462149:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829615.462159:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829615.462164:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280908800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829615.462177:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280908928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829615.462191:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829615.462196:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829615.462198:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829615.462204:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4165 00000100:00080000:2.0:1787829615.462218:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829615.462225:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829615.462231:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829615.462245:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280909056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829615.462257:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829615.462262:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280908928:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787829615.462263:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829615.462269:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829615.462276:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4165 00000100:00080000:2.0:1787829615.462284:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829615.462291:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280909056:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787829615.462292:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829615.462303:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829615.462320:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280909184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829615.462350:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829615.462355:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829615.462358:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829615.462363:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4165 00000100:00080000:2.0:1787829615.462368:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829615.462370:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280909184:192.168.203.124@tcp:8:kworker.0 00000100:00080000:2.0:1787829615.462377:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829615.462385:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829615.462400:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280909312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829615.462431:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280909312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829616.486080:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829617.510076:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829618.534014:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829618.534142:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787829618.534184:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829618.534200:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829618.534229:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829618.534234:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829618.534240:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829618.534248:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787829618.534334:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829615/real 1787829618] req@ffff8ab3f0439a40 x1874671280909312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829618.534366:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280909312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829618.534384:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829618.534390:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829618.534395:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280909312:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829618.534403:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829615/real 1787829618] req@ffff8ab3f043bc00 x1874671280909184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829618.534416:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280909184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829618.534424:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829618.534428:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829618.534431:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280909184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829618.534438:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829615/real 1787829618] req@ffff8ab3f04383c0 x1874671280908928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829618.534449:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280908928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829618.534458:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829618.534462:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829618.534465:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280908928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829618.534472:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829615/real 1787829618] req@ffff8ab3f043ad00 x1874671280909056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829618.534485:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280909056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829618.534494:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829618.534497:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829618.534501:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280909056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829618.534508:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829615/real 1787829618] req@ffff8ab3f04392c0 x1874671280908800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829618.534521:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280908800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829618.534529:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829618.534533:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829618.534536:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280908800:192.168.203.124@tcp:250:kworker.0 00010000:00010000:3.0:1787829619.558015:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829620.582032:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829620.582056:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829620.582062:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829620.582066:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829620.582075:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4170 00000100:00080000:3.0:1787829620.582083:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829620.582089:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829620.582119:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280909440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829620.582138:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829620.582140:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829620.582142:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829620.582144:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4170 00000100:00080000:3.0:1787829620.582147:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829620.582149:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829620.582161:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829620.582170:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280909568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829620.582179:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829620.582181:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280909440:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829620.582182:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829620.582187:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829620.582195:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4170 00000100:00080000:3.0:1787829620.582197:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829620.582200:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829620.582203:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829620.582213:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280909696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829620.582222:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829620.582226:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829620.582228:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829620.582233:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4170 00000100:00080000:3.0:1787829620.582248:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829620.582253:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829620.582259:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787829620.582261:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829620.582279:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280909824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829620.582290:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829620.582293:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829620.582296:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829620.582314:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4170 00000100:00080000:3.0:1787829620.582319:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829620.582325:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829620/real 1787829620] req@ffff8ab3d0a412c0 x1874671280909440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829620.582327:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829620.582337:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829620.582353:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280909440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829620.582355:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280909952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829620.582382:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829620.582391:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829620.582396:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280909440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829620.582424:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280909568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829620.582445:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829620.582464:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829620/real 1787829620] req@ffff8ab3d0a43840 x1874671280909568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829620.582478:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280909568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829620.582487:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829620.582490:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829620.582494:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280909568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829620.582500:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280909696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829620.582519:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829620.582532:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829620/real 1787829620] req@ffff8ab3d0a43480 x1874671280909696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829620.582543:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280909696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829620.582551:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829620.582555:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829620.582558:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280909696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829620.582564:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280909824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829620.582585:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829620.582597:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829620/real 1787829620] req@ffff8ab3d0a430c0 x1874671280909824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829620.582608:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280909824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829620.582618:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829620.582621:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829620.582624:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280909824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829620.582631:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280909952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829620.582650:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829620.582661:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829620/real 1787829620] req@ffff8ab3d0a41e00 x1874671280909952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829620.582673:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280909952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829620.582683:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829620.582686:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829620.582689:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280909952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829621.606111:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829622.630149:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829623.654064:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829624.678082:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829625.701999:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829625.702007:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829625.702010:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829625.702020:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4175 00000100:00080000:3.0:1787829625.702030:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829625.702036:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829625.702042:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829625.702070:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280910080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829625.702092:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829625.702094:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829625.702096:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829625.702099:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4175 00000100:00080000:3.0:1787829625.702101:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829625.702103:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829625.702111:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829625.702119:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280910208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829625.702127:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829625.702130:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829625.702131:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829625.702136:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280910080:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829625.702138:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4175 00000100:00080000:3.0:1787829625.702143:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829625.702147:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829625.702150:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829625.702160:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280910336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829625.702166:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829625.702168:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829625.702170:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829625.702172:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4175 00000100:00080000:3.0:1787829625.702187:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829625.702190:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829625.702193:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829625.702198:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829625.702206:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280910464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829625.702214:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829625.702216:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829625.702218:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829625.702220:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4175 00000100:00080000:3.0:1787829625.702222:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829625.702225:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829625.702229:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829625.702237:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280910592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829625.702241:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829625/real 1787829625] req@ffff8ab3d0a43c00 x1874671280910080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829625.702281:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280910080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829625.702292:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829625.702295:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829625.702298:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280910080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829625.702316:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280910208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829625.702333:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829625.702342:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829625/real 1787829625] req@ffff8ab3d0a40000 x1874671280910208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829625.702350:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280910208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829625.702356:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829625.702358:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829625.702360:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280910208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829625.702364:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280910336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829625.702374:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829625.702379:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829625/real 1787829625] req@ffff8ab3d0a421c0 x1874671280910336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829625.702387:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280910336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829625.702392:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829625.702393:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829625.702396:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280910336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829625.702400:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280910464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829625.702410:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829625.702415:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829625/real 1787829625] req@ffff8ab3d0a41e00 x1874671280910464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829625.702422:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280910464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829625.702428:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829625.702430:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829625.702431:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280910464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829625.702435:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280910592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829625.702444:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829625.702450:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829625/real 1787829625] req@ffff8ab3d0a430c0 x1874671280910592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829625.702457:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280910592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829625.702462:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829625.702464:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829625.702466:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280910592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829626.726192:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829627.750037:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829628.774047:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829629.798039:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829630.822008:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829630.822018:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829630.822022:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829630.822035:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4180 00000100:00080000:2.0:1787829630.822045:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829630.822060:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829630.822078:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829630.822107:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280910720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829630.822131:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829630.822133:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829630.822135:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829630.822137:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4180 00000100:00080000:2.0:1787829630.822140:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829630.822143:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829630.822153:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829630.822166:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280910848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829630.822167:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280910720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829630.822187:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829630.822189:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829630.822191:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829630.822194:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4180 00000100:00080000:2.0:1787829630.822196:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829630.822199:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829630.822202:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829630.822210:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280910976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829630.822216:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829630.822219:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829630.822220:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829630.822223:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4180 00000100:00080000:2.0:1787829630.822249:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829630.822253:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829630.822256:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829630.822264:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829630.822266:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280911104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829630.822274:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829630.822277:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829630.822279:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829630.822283:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4180 00000100:00080000:2.0:1787829630.822290:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829630.822293:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829630.822297:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829630.822304:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280911232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829630.822324:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829630/real 1787829630] req@ffff8ab3f0439680 x1874671280910720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829630.822339:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280910720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829630.822349:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829630.822352:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829630.822355:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280910720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829630.822394:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280910848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829630.822418:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829630.822434:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829630/real 1787829630] req@ffff8ab3f043b0c0 x1874671280910848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829630.822443:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280910848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829630.822449:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829630.822452:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829630.822454:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280910848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829630.822458:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280910976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829630.822482:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829630.822488:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829630/real 1787829630] req@ffff8ab3f043a580 x1874671280910976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829630.822496:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280910976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829630.822506:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829630.822508:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829630.822510:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280910976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829630.822513:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280911104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829630.822524:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829630.822530:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829630/real 1787829630] req@ffff8ab3f0438f00 x1874671280911104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829630.822537:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280911104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829630.822545:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829630.822546:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829630.822548:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280911104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829630.822552:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280911232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829630.822561:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829630.822567:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829630/real 1787829630] req@ffff8ab3f0438b40 x1874671280911232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829630.822574:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280911232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829630.822581:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829630.822582:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829630.822585:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280911232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829631.846046:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829632.870076:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829633.894114:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829634.918076:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829635.942003:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829635.942011:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829635.942015:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829635.942024:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4185 00000100:00080000:0.0:1787829635.942034:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829635.942042:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829635.942063:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829635.942084:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280911360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829635.942114:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829635.942119:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829635.942121:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829635.942126:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4185 00000100:00080000:0.0:1787829635.942130:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829635.942136:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829635.942147:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829635.942152:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280911360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829635.942164:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280911488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829635.942177:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829635.942180:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829635.942182:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829635.942187:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4185 00000100:00080000:0.0:1787829635.942222:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829635.942228:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829635.942234:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787829635.942235:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829635.942252:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280911616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829635.942270:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829635.942274:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829635.942276:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829635.942281:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4185 00000100:00080000:0.0:1787829635.942285:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829635.942292:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829635.942293:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829635/real 1787829635] req@ffff8ab3f075da40 x1874671280911360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829635.942300:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829635.942314:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280911744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829635.942322:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280911360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829635.942324:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829635.942331:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829635.942335:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829635.942342:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4185 00000100:00080000:0.0:1787829635.942345:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829635.942347:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829635.942348:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829635.942357:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787829635.942358:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829635.942366:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280911360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829635.942372:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280911872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829635.942395:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280911488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829635.942423:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829635.942437:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829635/real 1787829635] req@ffff8ab3f075cb40 x1874671280911488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829635.942451:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280911488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829635.942460:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829635.942464:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829635.942468:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280911488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829635.942475:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280911616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829635.942511:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829635.942524:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829635/real 1787829635] req@ffff8ab3f075f0c0 x1874671280911616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829635.942538:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280911616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829635.942547:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829635.942551:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829635.942554:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280911616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829635.942561:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280911744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829635.942583:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829635.942595:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829635/real 1787829635] req@ffff8ab3f075de00 x1874671280911744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829635.942608:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280911744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829635.942618:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829635.942622:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829635.942626:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280911744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829635.942632:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280911872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829635.942650:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829635.942659:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829635/real 1787829635] req@ffff8ab3f075cf00 x1874671280911872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829635.942667:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280911872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829635.942673:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829635.942676:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829635.942678:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280911872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829636.965997:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829637.990075:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829639.014074:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829640.038090:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829641.062000:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829641.062009:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829641.062014:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829641.062022:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4190 00000100:00080000:0.0:1787829641.062031:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829641.062036:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829641.062063:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829641.062071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280912000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829641.062090:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829641.062093:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829641.062094:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829641.062097:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4190 00000100:00080000:0.0:1787829641.062099:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829641.062102:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829641.062109:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829641.062119:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280912128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829641.062127:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829641.062131:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829641.062134:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829641.062135:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280912000:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829641.062138:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4190 00000100:00080000:0.0:1787829641.062144:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829641.062147:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829641.062150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829641.062170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280912256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829641.062183:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829641.062186:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829641.062187:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829641.062190:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4190 00000100:00080000:0.0:1787829641.062195:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829641.062198:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829641.062202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829641.062205:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829641.062215:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280912384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829641.062223:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829641.062225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829641.062226:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829641.062229:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4190 00000100:00080000:0.0:1787829641.062231:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829641.062234:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829641.062238:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829641.062247:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829641/real 1787829641] req@ffff8ab3f075c000 x1874671280912000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829641.062248:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280912512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829641.062285:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280912000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829641.062295:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829641.062298:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829641.062301:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280912000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829641.062321:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280912128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829641.062336:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829641.062345:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829641/real 1787829641] req@ffff8ab3f075cf00 x1874671280912128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829641.062353:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280912128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829641.062359:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829641.062361:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829641.062363:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280912128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829641.062368:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280912256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829641.062378:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829641.062384:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829641/real 1787829641] req@ffff8ab3f075de00 x1874671280912256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829641.062391:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280912256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829641.062396:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829641.062398:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829641.062400:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280912256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829641.062403:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280912384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829641.062413:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829641.062419:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829641/real 1787829641] req@ffff8ab3f075f0c0 x1874671280912384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829641.062426:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280912384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829641.062432:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829641.062434:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829641.062436:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280912384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829641.062439:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280912512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829641.062448:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829641.062455:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829641/real 1787829641] req@ffff8ab3f075cb40 x1874671280912512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829641.062467:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280912512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829641.062477:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829641.062481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829641.062484:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280912512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829642.086074:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829643.110060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829644.134099:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829645.158092:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829646.182021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829646.182029:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829646.182033:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829646.182042:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4195 00000100:00080000:1.0:1787829646.182051:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829646.182057:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829646.182102:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280912640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829646.182142:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829646.182145:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829646.182175:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829646.182180:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4195 00000100:00080000:1.0:1787829646.182184:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829646.182188:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829646.182199:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829646.182214:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280912768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829646.182234:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280912640:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787829646.182236:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829646.182247:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829646.182250:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829646.182256:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4195 00000100:00080000:1.0:1787829646.182259:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829646.182263:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829646.182268:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829646.182281:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280912896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829646.182292:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829646.182295:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829646.182297:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829646.182301:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4195 00000100:00080000:1.0:1787829646.182305:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829646.182308:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829646.182313:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787829646.182335:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829646.182343:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280913024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829646.182350:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829646.182353:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829646.182354:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829646.182358:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4195 00000100:00080000:1.0:1787829646.182361:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829646.182363:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829646.182366:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829646.182374:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280913152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829646.182408:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829646/real 1787829646] req@ffff8ab3d22fed00 x1874671280912640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829646.182427:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280912640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829646.182437:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829646.182440:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829646.182444:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280912640:192.168.203.124@tcp:250:kworker.0 00010000:00010000:3.0:1787829646.182465:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829646.182467:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280912768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829646.182507:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829646.182522:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829646/real 1787829646] req@ffff8ab3d22fc780 x1874671280912768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829646.182534:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280912768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829646.182539:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829646.182542:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829646.182544:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280912768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829646.182549:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280912896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829646.182563:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829646.182572:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829646/real 1787829646] req@ffff8ab3d22fe1c0 x1874671280912896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829646.182584:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280912896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829646.182593:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829646.182596:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829646.182600:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280912896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829646.182606:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280913024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829646.182627:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829646.182635:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829646/real 1787829646] req@ffff8ab3d22fde00 x1874671280913024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829646.182648:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280913024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829646.182655:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829646.182658:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829646.182660:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280913024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829646.182664:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280913152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829646.182674:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829646.182680:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829646/real 1787829646] req@ffff8ab3d22fd2c0 x1874671280913152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829646.182687:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280913152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829646.182693:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829646.182695:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829646.182696:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280913152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829647.206088:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829648.230135:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829649.254041:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829650.278036:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829651.302007:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829651.302015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829651.302019:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829651.302027:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4200 00000100:00080000:0.0:1787829651.302036:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787829651.302040:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829651.302044:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829651.302077:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280913280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829651.302112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829651.302115:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829651.302116:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829651.302119:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4200 00000100:00080000:0.0:1787829651.302123:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829651.302126:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829651.302135:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829651.302141:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280913280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829651.302147:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280913408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829651.302156:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829651.302158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829651.302160:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829651.302162:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4200 00000100:00080000:0.0:1787829651.302164:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829651.302168:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829651.302172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829651.302181:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280913536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829651.302187:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829651.302190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829651.302191:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829651.302194:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4200 00000100:00080000:0.0:1787829651.302197:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829651.302200:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829651.302204:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829651.302208:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829651.302217:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280913664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829651.302223:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829651.302225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829651.302227:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829651.302230:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4200 00000100:00080000:0.0:1787829651.302233:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829651.302236:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829651.302240:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829651.302244:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829651/real 1787829651] req@ffff8ab3f075da40 x1874671280913280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829651.302251:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280913792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829651.302277:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280913280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829651.302285:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829651.302288:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829651.302291:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280913280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829651.302308:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280913408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829651.302322:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829651.302331:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829651/real 1787829651] req@ffff8ab3f075e940 x1874671280913408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829651.302339:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280913408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829651.302344:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829651.302346:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829651.302348:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280913408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829651.302352:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280913536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829651.302361:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829651.302366:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829651/real 1787829651] req@ffff8ab3f075e580 x1874671280913536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829651.302373:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280913536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829651.302383:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829651.302385:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829651.302387:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280913536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829651.302390:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280913664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829651.302399:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829651.302404:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829651/real 1787829651] req@ffff8ab3f075ed00 x1874671280913664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829651.302411:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280913664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829651.302417:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829651.302419:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829651.302421:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280913664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829651.302424:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280913792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829651.302433:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829651.302438:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829651/real 1787829651] req@ffff8ab3f075f840 x1874671280913792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829651.302445:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280913792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829651.302451:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829651.302453:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829651.302455:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280913792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829652.326094:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829653.350043:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829654.374092:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829655.398102:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829656.422024:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829656.422034:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829656.422038:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829656.422047:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4206 00000100:00080000:0.0:1787829656.422069:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829656.422076:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829656.422108:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829656.422114:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280913920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829656.422138:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829656.422141:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829656.422142:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829656.422145:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4206 00000100:00080000:0.0:1787829656.422147:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829656.422150:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829656.422158:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829656.422168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280914048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829656.422177:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829656.422179:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829656.422183:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829656.422184:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280913920:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829656.422187:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4206 00000100:00080000:0.0:1787829656.422193:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829656.422197:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829656.422200:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829656.422209:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280914176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829656.422215:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829656.422217:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829656.422219:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829656.422222:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4206 00000100:00080000:0.0:1787829656.422234:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829656.422237:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829656.422241:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:2.0:1787829656.422244:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829656.422253:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280914304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829656.422261:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829656.422264:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829656.422265:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829656.422267:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4206 00000100:00080000:0.0:1787829656.422270:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829656.422272:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829656.422275:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829656.422284:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280914432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829656.422291:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829656/real 1787829656] req@ffff8ab3f075f480 x1874671280913920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829656.422308:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280913920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829656.422317:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829656.422320:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829656.422323:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280913920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829656.422341:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280914048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829656.422357:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829656.422366:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829656/real 1787829656] req@ffff8ab3f075f840 x1874671280914048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829656.422375:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280914048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829656.422381:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829656.422384:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829656.422386:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280914048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829656.422389:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280914176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787829656.422401:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829656.422409:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829656/real 1787829656] req@ffff8ab3f075ed00 x1874671280914176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829656.422422:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280914176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829656.422431:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829656.422434:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829656.422438:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280914176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829656.422444:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280914304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829656.422464:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829656.422473:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829656/real 1787829656] req@ffff8ab3f075e580 x1874671280914304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829656.422482:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280914304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829656.422489:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829656.422491:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829656.422493:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280914304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829656.422497:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280914432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787829656.422509:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829656.422519:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829656/real 1787829656] req@ffff8ab3f075e940 x1874671280914432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829656.422539:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280914432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829656.422550:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829656.422553:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829656.422557:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280914432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829657.446128:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829658.470067:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829659.494148:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829660.518068:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829661.541971:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829661.541982:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829661.541987:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829661.541999:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4211 00000100:00080000:2.0:1787829661.542011:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829661.542033:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829661.542040:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829661.542080:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280914560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829661.542108:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829661.542111:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829661.542113:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829661.542115:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4211 00000100:00080000:2.0:1787829661.542119:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829661.542124:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829661.542135:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829661.542149:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280914560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829661.542151:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280914688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829661.542174:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829661.542176:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829661.542178:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829661.542183:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4211 00000100:00080000:2.0:1787829661.542185:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829661.542188:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829661.542193:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829661.542204:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280914816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829661.542210:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829661.542212:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829661.542215:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829661.542219:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4211 00000100:00080000:2.0:1787829661.542222:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829661.542228:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829661.542232:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829661.542237:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829661.542251:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280914944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829661.542259:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829661.542262:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829661.542264:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829661.542271:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4211 00000100:00080000:2.0:1787829661.542276:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829661.542280:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829661.542284:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829661.542293:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829661/real 1787829661] req@ffff8ab3f0438780 x1874671280914560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829661.542293:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280915072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829661.542344:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280914560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829661.542362:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829661.542366:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829661.542369:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280914560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829661.542389:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280914688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829661.542409:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829661.542418:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829661/real 1787829661] req@ffff8ab3f0438b40 x1874671280914688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829661.542426:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280914688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829661.542432:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829661.542434:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829661.542437:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280914688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829661.542441:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280914816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829661.542452:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829661.542458:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829661/real 1787829661] req@ffff8ab3f0438f00 x1874671280914816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829661.542465:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280914816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829661.542470:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829661.542472:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829661.542473:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280914816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829661.542477:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280914944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829661.542487:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829661.542493:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829661/real 1787829661] req@ffff8ab3f043a580 x1874671280914944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829661.542499:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280914944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829661.542505:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829661.542507:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829661.542509:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280914944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829661.542513:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280915072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829661.542522:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829661.542528:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829661/real 1787829661] req@ffff8ab3f043b0c0 x1874671280915072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829661.542535:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280915072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829661.542541:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829661.542543:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829661.542545:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280915072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829662.566089:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829663.590098:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829664.614116:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829665.638153:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829666.662027:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829666.662038:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829666.662043:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829666.662053:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4216 00000100:00080000:2.0:1787829666.662062:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829666.662069:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829666.662090:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829666.662108:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280915200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829666.662136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829666.662139:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829666.662141:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829666.662144:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4216 00000100:00080000:2.0:1787829666.662148:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829666.662152:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829666.662162:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829666.662168:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280915200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829666.662176:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280915328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829666.662184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829666.662188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829666.662189:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829666.662192:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4216 00000100:00080000:2.0:1787829666.662195:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829666.662197:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829666.662200:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829666.662210:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280915456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829666.662216:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829666.662219:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829666.662220:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829666.662224:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4216 00000800:00000100:1.0:1787829666.662228:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829666.662230:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829666.662235:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829666.662240:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829666.662249:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280915584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829666.662255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829666.662258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829666.662259:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829666.662263:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4216 00000100:00080000:2.0:1787829666.662266:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829666.662268:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829666.662273:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829666/real 1787829666] req@ffff8ab3f0439680 x1874671280915200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829666.662276:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829666.662292:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280915200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829666.662297:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280915712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829666.662307:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829666.662315:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829666.662320:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280915200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829666.662338:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280915328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829666.662352:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829666.662360:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829666/real 1787829666] req@ffff8ab3f04392c0 x1874671280915328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829666.662368:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280915328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829666.662378:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829666.662380:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829666.662382:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280915328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829666.662386:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280915456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829666.662395:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829666.662401:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829666/real 1787829666] req@ffff8ab3f043ad00 x1874671280915456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829666.662408:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280915456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829666.662413:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829666.662415:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829666.662417:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280915456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829666.662420:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280915584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829666.662430:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829666.662435:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829666/real 1787829666] req@ffff8ab3f04383c0 x1874671280915584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829666.662442:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280915584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829666.662448:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829666.662450:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829666.662452:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280915584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829666.662456:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280915712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829666.662465:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829666.662470:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829666/real 1787829666] req@ffff8ab3f043bc00 x1874671280915712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829666.662476:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280915712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829666.662482:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829666.662484:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829666.662486:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280915712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829667.686056:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829668.710045:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829669.734112:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829670.758103:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829671.782042:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829671.782053:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829671.782058:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829671.782069:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4221 00000100:00080000:0.0:1787829671.782081:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829671.782087:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829671.782093:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829671.782123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280915840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829671.782145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829671.782147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829671.782148:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829671.782151:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4221 00000100:00080000:0.0:1787829671.782154:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829671.782157:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829671.782164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829671.782173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280915968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829671.782182:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829671.782185:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829671.782186:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829671.782191:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4221 00000100:00100000:1.0:1787829671.782193:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280915840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829671.782194:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829671.782201:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829671.782208:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829671.782223:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280916096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829671.782235:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829671.782239:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829671.782241:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829671.782246:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4221 00000100:00080000:0.0:1787829671.782259:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829671.782266:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829671.782267:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829671.782275:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829671.782287:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280916224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829671.782294:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829671.782297:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829671.782299:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829671.782303:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4221 00000100:00080000:0.0:1787829671.782307:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829671.782312:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829671.782318:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829671.782324:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829671/real 1787829671] req@ffff8ab3f075da40 x1874671280915840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829671.782335:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280916352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829671.782351:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280915840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829671.782365:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829671.782370:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829671.782375:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280915840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829671.782399:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280915968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829671.782425:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829671.782442:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829671/real 1787829671] req@ffff8ab3f075cb40 x1874671280915968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829671.782456:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280915968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829671.782467:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829671.782471:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829671.782475:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280915968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829671.782482:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280916096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829671.782503:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829671.782516:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829671/real 1787829671] req@ffff8ab3f075f0c0 x1874671280916096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829671.782529:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280916096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829671.782538:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829671.782541:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829671.782545:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280916096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829671.782552:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280916224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829671.782572:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829671.782584:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829671/real 1787829671] req@ffff8ab3f075de00 x1874671280916224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829671.782597:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280916224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829671.782608:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829671.782611:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829671.782615:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280916224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829671.782621:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280916352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829671.782642:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829671.782654:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829671/real 1787829671] req@ffff8ab3f075cf00 x1874671280916352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829671.782667:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280916352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829671.782678:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829671.782681:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829671.782684:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280916352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829672.806179:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829673.830095:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829674.854127:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829675.878061:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829676.902042:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829676.902052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829676.902058:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829676.902068:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4226 00000100:00080000:0.0:1787829676.902080:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829676.902089:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829676.902099:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829676.902129:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280916480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829676.902157:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829676.902161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829676.902164:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829676.902168:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4226 00000100:00080000:0.0:1787829676.902175:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829676.902180:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829676.902189:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829676.902208:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280916608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829676.902212:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280916480:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829676.902221:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829676.902231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829676.902236:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829676.902242:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4226 00000100:00080000:0.0:1787829676.902245:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829676.902250:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829676.902257:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829676.902270:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280916736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829676.902279:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829676.902283:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829676.902285:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829676.902290:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4226 00000100:00080000:0.0:1787829676.902304:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829676.902310:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829676.902315:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829676.902318:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829676.902334:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280916864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829676.902346:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829676.902350:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829676.902352:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829676.902357:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4226 00000100:00080000:0.0:1787829676.902360:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829676.902364:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829676.902370:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829676.902383:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280916992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829676.902401:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829676/real 1787829676] req@ffff8ab3f075c000 x1874671280916480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829676.902426:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280916480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829676.902440:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829676.902446:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829676.902451:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280916480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829676.902485:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280916608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829676.902513:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829676.902530:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829676/real 1787829676] req@ffff8ab3f075cf00 x1874671280916608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829676.902544:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280916608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829676.902554:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829676.902557:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829676.902560:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280916608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829676.902565:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280916736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829676.902582:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829676.902591:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829676/real 1787829676] req@ffff8ab3f075de00 x1874671280916736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829676.902598:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280916736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829676.902605:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829676.902607:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829676.902609:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280916736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829676.902613:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280916864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829676.902636:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829676.902645:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829676/real 1787829676] req@ffff8ab3f075f0c0 x1874671280916864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829676.902658:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280916864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829676.902670:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829676.902674:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829676.902677:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280916864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829676.902684:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280916992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829676.902707:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829676.902720:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829676/real 1787829676] req@ffff8ab3f075cb40 x1874671280916992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829676.902732:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280916992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829676.902744:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829676.902748:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829676.902751:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280916992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829677.926070:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829678.950060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829679.974061:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829680.998087:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829682.022098:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829682.022110:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829682.022114:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1787829682.022125:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829682.022126:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4231 00000100:00080000:1.0:1787829682.022139:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829682.022147:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829682.022201:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280917120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829682.022249:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829682.022254:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829682.022256:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829682.022262:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4231 00000100:00080000:1.0:1787829682.022266:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829682.022272:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829682.022284:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829682.022296:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280917120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829682.022305:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280917248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829682.022322:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829682.022327:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829682.022329:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829682.022335:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4231 00000100:00080000:1.0:1787829682.022359:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829682.022377:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829682.022384:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829682.022399:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280917376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829682.022409:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829682.022413:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829682.022416:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829682.022421:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4231 00000100:00080000:1.0:1787829682.022425:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829682.022429:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829682.022445:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829682.022456:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280917504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829682.022465:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829682.022466:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280917248:192.168.203.124@tcp:38:kworker.0 00000100:00080000:1.0:1787829682.022472:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829682.022476:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829682.022485:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4231 00000100:00080000:1.0:1787829682.022487:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829682.022492:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787829682.022496:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829682.022510:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280917632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829682.022534:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280917376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829682.022600:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280917504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829682.022618:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280917632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829683.046086:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829684.070067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829685.094091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829685.094219:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787829685.094263:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829685.094279:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829685.094324:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829685.094336:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829685.094353:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829685.094370:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:2.0:1787829685.094379:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829682/real 1787829685] req@ffff8ab3d22fde00 x1874671280917248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829685.094408:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280917248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829685.094417:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829685.094421:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829685.094425:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280917248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829685.094432:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829682/real 1787829685] req@ffff8ab3d22fe1c0 x1874671280917376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829685.094441:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280917376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829685.094446:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829685.094449:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829685.094451:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280917376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829685.094455:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829682/real 1787829685] req@ffff8ab3d22fd2c0 x1874671280917120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829685.094463:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280917120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829685.094468:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829685.094471:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829685.094473:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280917120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829685.094498:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829682/real 1787829685] req@ffff8ab3d22fc780 x1874671280917504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829685.094511:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280917504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829685.094521:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829685.094526:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829685.094529:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280917504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787829685.094536:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829682/real 1787829685] req@ffff8ab3d22fed00 x1874671280917632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829685.094548:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280917632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829685.094557:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829685.094561:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787829685.094564:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280917632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829686.118059:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829687.142065:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829687.142073:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829687.142078:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829687.142090:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4236 00010000:00010000:3.0:1787829687.142099:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829687.142101:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829687.142107:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829687.142143:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280917760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829687.142172:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829687.142175:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829687.142176:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829687.142179:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4236 00000100:00080000:2.0:1787829687.142182:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829687.142185:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829687.142193:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829687.142201:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280917888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829687.142211:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829687.142213:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829687.142215:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829687.142223:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4236 00000100:00100000:1.0:1787829687.142225:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280917760:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829687.142226:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829687.142234:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829687.142247:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829687.142257:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280918016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829687.142267:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829687.142271:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829687.142273:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829687.142278:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4236 00000100:00080000:2.0:1787829687.142283:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829687.142288:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829687.142292:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829687.142305:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280918144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829687.142314:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829687.142317:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829687.142319:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829687.142323:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4236 00000100:00080000:2.0:1787829687.142331:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829687.142336:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829687.142341:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787829687.142349:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829687.142358:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280918272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829687.142432:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829687/real 1787829687] req@ffff8ab3f0439a40 x1874671280917760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829687.142457:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280917760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829687.142478:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829687.142483:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829687.142489:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280917760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829687.142523:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280917888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829687.142553:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829687.142590:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829687/real 1787829687] req@ffff8ab3f043a940 x1874671280917888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829687.142606:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280917888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829687.142615:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829687.142620:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829687.142623:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280917888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829687.142631:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280918016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829687.142654:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829687.142667:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829687/real 1787829687] req@ffff8ab3f043a1c0 x1874671280918016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829687.142680:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280918016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829687.142690:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829687.142693:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829687.142697:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280918016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829687.142705:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280918144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829687.142730:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829687.142745:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829687/real 1787829687] req@ffff8ab3f0439e00 x1874671280918144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829687.142759:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280918144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829687.142772:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829687.142777:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829687.142781:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280918144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829687.142789:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280918272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829687.142813:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829687.142827:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829687/real 1787829687] req@ffff8ab3f043b480 x1874671280918272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829687.142862:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280918272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829687.142891:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829687.142895:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829687.142899:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280918272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829688.166087:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829689.190130:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829690.214025:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829691.238085:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829692.262011:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829692.262020:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829692.262024:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829692.262035:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4241 00000100:00080000:2.0:1787829692.262047:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829692.262054:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829692.262057:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829692.262089:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280918400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829692.262110:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829692.262113:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829692.262114:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829692.262117:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4241 00000100:00080000:2.0:1787829692.262120:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829692.262123:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829692.262131:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829692.262142:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280918528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829692.262153:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280918400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829692.262155:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829692.262170:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829692.262171:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829692.262175:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4241 00000100:00080000:2.0:1787829692.262177:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829692.262179:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829692.262182:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829692.262194:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280918656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829692.262205:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829692.262208:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829692.262210:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829692.262215:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4241 00000100:00080000:2.0:1787829692.262230:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829692.262236:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829692.262242:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829692.262245:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829692.262262:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280918784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829692.262270:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829692.262272:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829692.262274:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829692.262277:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4241 00000100:00080000:2.0:1787829692.262281:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829692.262285:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829692.262290:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829692.262299:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829692/real 1787829692] req@ffff8ab3f043b840 x1874671280918400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829692.262300:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280918912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829692.262366:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280918400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829692.262377:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829692.262381:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829692.262385:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280918400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829692.262408:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280918528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829692.262425:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829692.262434:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829692/real 1787829692] req@ffff8ab3f043b480 x1874671280918528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829692.262442:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280918528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829692.262448:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829692.262450:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829692.262452:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280918528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829692.262456:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280918656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829692.262466:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829692.262472:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829692/real 1787829692] req@ffff8ab3f0439e00 x1874671280918656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829692.262479:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280918656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829692.262484:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829692.262486:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829692.262489:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280918656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829692.262493:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280918784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829692.262504:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829692.262509:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829692/real 1787829692] req@ffff8ab3f043a1c0 x1874671280918784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829692.262516:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280918784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829692.262524:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829692.262526:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829692.262528:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280918784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829692.262532:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280918912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829692.262542:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829692.262547:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829692/real 1787829692] req@ffff8ab3f043a940 x1874671280918912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829692.262554:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280918912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829692.262561:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829692.262562:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829692.262564:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280918912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829693.286041:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829694.310086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829695.334072:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829696.358152:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829697.381994:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829697.382006:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829697.382010:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829697.382020:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4247 00000100:00080000:2.0:1787829697.382031:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829697.382038:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829697.382072:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829697.382084:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280919040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829697.382128:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829697.382132:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829697.382135:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829697.382140:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4247 00000100:00080000:2.0:1787829697.382144:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829697.382149:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829697.382162:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829697.382185:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280919040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829697.382186:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280919168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829697.382205:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829697.382209:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829697.382212:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829697.382218:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4247 00000100:00080000:2.0:1787829697.382224:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829697.382230:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829697.382237:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829697.382255:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280919296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829697.382267:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829697.382273:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787829697.382278:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829697.382278:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829697.382290:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4247 00000100:00080000:2.0:1787829697.382296:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829697.382321:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829697.382327:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829697.382341:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280919424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829697.382349:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829697.382353:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829697.382354:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829697.382358:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829697/real 1787829697] req@ffff8ab3f0439a40 x1874671280919040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829697.382361:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4247 00000100:00080000:2.0:1787829697.382367:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829697.382374:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829697.382379:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829697.382384:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280919040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829697.382390:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280919552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829697.382397:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829697.382406:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829697.382413:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280919040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829697.382439:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280919168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829697.382456:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829697.382465:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829697/real 1787829697] req@ffff8ab3f043bc00 x1874671280919168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829697.382473:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280919168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829697.382479:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829697.382481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829697.382483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280919168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829697.382487:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280919296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829697.382497:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829697.382502:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829697/real 1787829697] req@ffff8ab3f04383c0 x1874671280919296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829697.382509:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280919296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829697.382519:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829697.382521:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829697.382523:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280919296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829697.382527:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280919424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829697.382536:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829697.382542:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829697/real 1787829697] req@ffff8ab3f043ad00 x1874671280919424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829697.382549:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280919424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829697.382556:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829697.382558:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829697.382559:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280919424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829697.382563:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280919552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829697.382573:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829697.382578:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829697/real 1787829697] req@ffff8ab3f04392c0 x1874671280919552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829697.382585:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280919552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829697.382591:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829697.382593:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829697.382596:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280919552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829698.406104:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829699.430084:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829700.454109:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829701.478026:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829702.502007:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829702.502016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829702.502019:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829702.502029:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4252 00000100:00080000:0.0:1787829702.502040:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829702.502048:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829702.502059:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829702.502088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280919680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829702.502112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829702.502115:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829702.502116:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829702.502119:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4252 00000100:00080000:0.0:1787829702.502122:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829702.502126:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829702.502135:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829702.502142:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280919680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829702.502147:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280919808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829702.502157:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829702.502159:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829702.502161:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829702.502164:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4252 00000100:00080000:0.0:1787829702.502166:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829702.502168:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829702.502172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829702.502180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280919936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829702.502187:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829702.502190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829702.502191:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829702.502194:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4252 00000100:00080000:0.0:1787829702.502196:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829702.502199:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829702.502204:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787829702.502206:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829702.502216:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280920064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829702.502222:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829702.502225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829702.502226:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829702.502228:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4252 00000100:00080000:0.0:1787829702.502230:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829702.502233:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829702.502236:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829702.502245:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280920192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829702.502252:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829702/real 1787829702] req@ffff8ab3f075da40 x1874671280919680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829702.502267:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280919680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829702.502275:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829702.502279:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829702.502281:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280919680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829702.502299:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280919808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829702.502312:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829702.502321:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829702/real 1787829702] req@ffff8ab3f075e940 x1874671280919808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829702.502329:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280919808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829702.502334:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829702.502336:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829702.502338:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280919808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829702.502342:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280919936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829702.502352:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829702.502358:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829702/real 1787829702] req@ffff8ab3f075e580 x1874671280919936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829702.502365:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280919936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829702.502371:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829702.502372:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829702.502374:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280919936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829702.502378:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280920064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829702.502388:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829702.502393:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829702/real 1787829702] req@ffff8ab3f075ed00 x1874671280920064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829702.502400:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280920064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829702.502407:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829702.502409:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829702.502410:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280920064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829702.502414:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280920192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829702.502423:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829702.502428:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829702/real 1787829702] req@ffff8ab3f075f840 x1874671280920192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829702.502435:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280920192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829702.502442:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829702.502444:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829702.502446:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280920192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829703.526136:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829704.550091:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829705.574019:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829706.598062:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829707.621988:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829707.621999:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829707.622006:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829707.622017:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4257 00000100:00080000:0.0:1787829707.622029:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1787829707.622036:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829707.622037:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829707.622087:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280920320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829707.622111:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829707.622114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829707.622115:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829707.622118:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4257 00000100:00080000:0.0:1787829707.622120:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829707.622123:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829707.622132:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829707.622145:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280920448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829707.622156:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829707.622162:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829707.622163:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280920320:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829707.622165:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829707.622182:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4257 00000100:00080000:0.0:1787829707.622187:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829707.622193:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829707.622200:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829707.622219:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280920576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829707.622238:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829707.622243:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829707.622250:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829707.622253:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829707.622262:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4257 00000100:00080000:0.0:1787829707.622267:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829707.622273:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829707.622281:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829707.622295:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280920704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829707.622306:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829707/real 1787829707] req@ffff8ab3f075f480 x1874671280920320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829707.622309:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829707.622331:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829707.622334:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829707.622343:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4257 00000100:00100000:1.0:1787829707.622344:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280920320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829707.622351:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829707.622358:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829707.622367:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787829707.622371:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829707.622380:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280920832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829707.622381:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829707.622399:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280920320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829707.622445:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280920448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829707.622468:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829707.622478:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829707/real 1787829707] req@ffff8ab3f075f840 x1874671280920448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829707.622487:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280920448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829707.622493:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829707.622496:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829707.622498:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280920448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829707.622503:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280920576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829707.622514:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829707.622520:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829707/real 1787829707] req@ffff8ab3f075ed00 x1874671280920576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829707.622528:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280920576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829707.622533:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829707.622535:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829707.622537:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280920576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829707.622540:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280920704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829707.622550:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829707.622556:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829707/real 1787829707] req@ffff8ab3f075e580 x1874671280920704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829707.622563:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280920704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829707.622570:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829707.622572:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829707.622574:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280920704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829707.622578:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280920832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829707.622588:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829707.622594:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829707/real 1787829707] req@ffff8ab3f075e940 x1874671280920832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829707.622601:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280920832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829707.622608:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829707.622609:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829707.622611:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280920832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829708.646108:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829709.670071:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829710.694102:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829711.718104:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829712.742022:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829712.742034:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829712.742038:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829712.742050:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4262 00000100:00080000:0.0:1787829712.742062:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829712.742072:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829712.742084:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829712.742114:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280920960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829712.742150:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829712.742154:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829712.742156:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829712.742161:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4262 00000100:00080000:0.0:1787829712.742165:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829712.742170:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829712.742181:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829712.742197:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280921088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829712.742205:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280920960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829712.742211:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829712.742217:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829712.742219:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829712.742226:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4262 00000100:00080000:0.0:1787829712.742231:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829712.742236:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829712.742242:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829712.742256:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280921216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829712.742263:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829712.742273:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829712.742278:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829712.742281:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829712.742287:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4262 00000100:00080000:0.0:1787829712.742292:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829712.742297:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829712.742304:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829712.742306:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829712/real 1787829712] req@ffff8ab3f075da40 x1874671280920960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829712.742321:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280921344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829712.742323:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280920960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829712.742353:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829712.742355:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829712.742359:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787829712.742363:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829712.742364:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280920960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829712.742368:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829712.742376:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4262 00000100:00080000:0.0:1787829712.742380:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829712.742386:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280921088:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787829712.742393:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829712.742400:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787829712.742406:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829712.742421:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280921472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829712.742421:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829712/real 1787829712] req@ffff8ab3f075cb40 x1874671280921088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829712.742461:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280921088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829712.742473:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829712.742476:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829712.742479:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280921088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829712.742484:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280921216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829712.742495:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829712.742502:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829712/real 1787829712] req@ffff8ab3f075f0c0 x1874671280921216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829712.742510:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280921216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829712.742515:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829712.742517:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829712.742520:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280921216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829712.742527:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280921344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829712.742548:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829712.742561:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829712/real 1787829712] req@ffff8ab3f075de00 x1874671280921344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829712.742575:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280921344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829712.742585:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829712.742589:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829712.742593:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280921344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829712.742610:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280921472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829712.742632:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829712.742645:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829712/real 1787829712] req@ffff8ab3f075cf00 x1874671280921472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829712.742672:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280921472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829712.742684:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829712.742688:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829712.742692:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280921472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829713.766069:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829714.790103:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829715.814079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829716.838028:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829717.862015:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829717.862032:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829717.862037:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829717.862048:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4267 00000100:00080000:2.0:1787829717.862059:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829717.862068:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829717.862090:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829717.862115:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280921600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829717.862139:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829717.862142:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829717.862144:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829717.862147:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4267 00000100:00080000:2.0:1787829717.862149:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829717.862152:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829717.862162:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829717.862174:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280921728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829717.862182:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280921600:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829717.862197:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829717.862200:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829717.862202:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829717.862205:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4267 00000100:00080000:2.0:1787829717.862208:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829717.862211:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829717.862214:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829717.862222:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280921856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829717.862229:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829717.862232:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829717.862233:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829717.862236:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4267 00000100:00080000:2.0:1787829717.862256:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829717.862259:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829717.862263:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829717.862268:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829717.862275:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280921984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829717.862282:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829717.862285:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829717.862287:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829717.862290:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4267 00000100:00080000:2.0:1787829717.862297:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829717.862300:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829717.862302:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829717.862310:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280922112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829717.862334:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829717/real 1787829717] req@ffff8ab3f0439680 x1874671280921600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829717.862356:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280921600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829717.862368:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829717.862373:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829717.862378:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280921600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829717.862404:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280921728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829717.862427:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829717.862442:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829717/real 1787829717] req@ffff8ab3f043b0c0 x1874671280921728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829717.862456:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280921728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829717.862465:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829717.862468:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829717.862472:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280921728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829717.862478:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280921856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829717.862497:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829717.862508:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829717/real 1787829717] req@ffff8ab3f043a580 x1874671280921856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829717.862522:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280921856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829717.862540:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829717.862547:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829717.862551:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280921856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829717.862558:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280921984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829717.862578:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829717.862590:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829717/real 1787829717] req@ffff8ab3f0438f00 x1874671280921984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829717.862603:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280921984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829717.862615:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829717.862618:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829717.862622:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280921984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829717.862643:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280922112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829717.862664:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829717.862677:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829717/real 1787829717] req@ffff8ab3f0438b40 x1874671280922112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829717.862690:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280922112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829717.862700:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829717.862704:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829717.862708:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280922112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829718.886051:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829719.910079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829720.934046:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829721.958073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829722.981982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829722.981993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829722.981998:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829722.982009:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4272 00000100:00080000:2.0:1787829722.982021:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829722.982030:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829722.982046:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829722.982070:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280922240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829722.982093:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829722.982096:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829722.982098:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829722.982100:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4272 00000100:00080000:2.0:1787829722.982104:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829722.982108:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829722.982116:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829722.982128:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280922240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829722.982136:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280922368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829722.982151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829722.982153:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829722.982155:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829722.982158:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4272 00000100:00080000:2.0:1787829722.982161:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829722.982163:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829722.982167:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829722.982175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280922496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829722.982182:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829722.982184:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829722.982186:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829722.982188:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4272 00000100:00080000:2.0:1787829722.982191:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829722.982195:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829722.982197:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829722.982202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829722.982210:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280922624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829722.982217:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829722.982219:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829722.982220:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829722.982223:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4272 00000100:00080000:2.0:1787829722.982226:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829722.982229:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829722.982233:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829722.982243:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829722/real 1787829722] req@ffff8ab3f0438780 x1874671280922240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829722.982244:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280922752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829722.982277:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280922240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829722.982286:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829722.982290:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829722.982293:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280922240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829722.982312:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280922368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829722.982329:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829722.982338:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829722/real 1787829722] req@ffff8ab3f0438b40 x1874671280922368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829722.982346:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280922368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829722.982353:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829722.982355:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829722.982357:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280922368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829722.982361:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280922496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829722.982371:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829722.982376:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829722/real 1787829722] req@ffff8ab3f0438f00 x1874671280922496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829722.982384:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280922496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829722.982389:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829722.982391:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829722.982393:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280922496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829722.982397:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280922624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829722.982408:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829722.982414:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829722/real 1787829722] req@ffff8ab3f043a580 x1874671280922624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829722.982422:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280922624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829722.982428:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829722.982430:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829722.982432:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280922624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829722.982436:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280922752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829722.982446:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829722.982451:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829722/real 1787829722] req@ffff8ab3f043b0c0 x1874671280922752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829722.982459:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280922752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829722.982465:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829722.982467:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829722.982469:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280922752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829724.006412:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829725.030096:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829726.054110:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829727.078041:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829728.101986:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829728.101994:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829728.101998:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829728.102008:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4277 00000100:00080000:2.0:1787829728.102018:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829728.102026:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829728.102057:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280922880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829728.102081:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829728.102085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829728.102088:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1787829728.102092:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829728.102094:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4277 00000100:00080000:2.0:1787829728.102102:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829728.102108:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829728.102119:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829728.102126:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280922880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829728.102139:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280923008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829728.102150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829728.102155:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829728.102157:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829728.102161:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4277 00000100:00080000:2.0:1787829728.102164:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829728.102167:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829728.102172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829728.102182:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280923136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829728.102190:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829728.102192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829728.102193:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829728.102196:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4277 00000100:00080000:2.0:1787829728.102207:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829728.102210:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829728.102213:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829728.102222:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829728.102223:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280923264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829728.102238:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829728.102242:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829728.102244:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829728.102248:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4277 00000100:00080000:2.0:1787829728.102251:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829728.102257:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829728.102263:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829728.102277:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280923392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829728.102313:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829728/real 1787829728] req@ffff8ab3f0439680 x1874671280922880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829728.102346:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280922880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829728.102362:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829728.102367:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829728.102372:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280922880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829728.102404:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280923008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829728.102433:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829728.102450:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829728/real 1787829728] req@ffff8ab3f04392c0 x1874671280923008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829728.102464:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280923008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829728.102473:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829728.102477:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829728.102481:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280923008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829728.102489:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280923136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829728.102513:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829728.102527:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829728/real 1787829728] req@ffff8ab3f043ad00 x1874671280923136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829728.102540:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280923136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829728.102549:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829728.102568:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829728.102573:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280923136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829728.102581:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280923264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829728.102605:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829728.102618:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829728/real 1787829728] req@ffff8ab3f04383c0 x1874671280923264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829728.102630:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280923264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829728.102652:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829728.102655:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829728.102659:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280923264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829728.102667:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280923392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829728.102689:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829728.102713:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829728/real 1787829728] req@ffff8ab3f043bc00 x1874671280923392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829728.102726:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280923392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829728.102737:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829728.102740:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829728.102744:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280923392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829729.126029:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829730.150092:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829731.174034:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829732.198073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829733.222042:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829733.222057:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829733.222077:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829733.222091:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4282 00010000:00010000:3.0:1787829733.222103:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829733.222119:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829733.222145:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829733.222202:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280923520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829733.222256:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829733.222261:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829733.222263:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829733.222274:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280923520:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829733.222287:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4282 00000100:00080000:2.0:1787829733.222292:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829733.222297:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829733.222310:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829733.222329:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280923648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829733.222352:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829733.222358:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829733.222362:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829733.222364:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829733.222369:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4282 00000100:00080000:2.0:1787829733.222384:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829733.222389:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829733.222395:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829733.222409:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829733/real 1787829733] req@ffff8ab3f0439a40 x1874671280923520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829733.222410:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280923776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829733.222427:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280923520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829733.222430:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829733.222438:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829733.222442:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829733.222445:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829733.222450:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829733.222453:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280923520:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829733.222458:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4282 00000100:00080000:2.0:1787829733.222461:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829733.222465:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829733.222469:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829733.222480:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280923648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829733.222481:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280923904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829733.222501:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829733.222503:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829733.222508:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829733.222509:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829733.222515:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829733/real 1787829733] req@ffff8ab3f043a940 x1874671280923648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829733.222515:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4282 00000100:00080000:2.0:1787829733.222540:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829733.222546:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829733.222551:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829733.222556:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280923648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829733.222562:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280924032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829733.222567:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829733.222574:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829733.222577:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280923648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829733.222582:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280923776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829733.222607:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829733.222615:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829733/real 1787829733] req@ffff8ab3f043a1c0 x1874671280923776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829733.222624:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280923776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829733.222634:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829733.222636:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829733.222638:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280923776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829733.222647:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280923904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829733.222659:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829733.222665:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829733/real 1787829733] req@ffff8ab3f0439e00 x1874671280923904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829733.222672:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280923904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829733.222679:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829733.222680:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829733.222682:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280923904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829733.222686:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280924032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829733.222695:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829733.222700:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829733/real 1787829733] req@ffff8ab3f043b480 x1874671280924032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829733.222707:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280924032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829733.222714:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829733.222715:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829733.222718:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280924032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829734.246089:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829735.270065:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829736.294121:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829737.318044:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829738.342015:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829738.342031:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829738.342038:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829738.342050:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4287 00010000:00010000:3.0:1787829738.342062:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829738.342063:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829738.342072:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829738.342123:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280924160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829738.342148:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829738.342151:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829738.342152:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829738.342155:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4287 00000100:00080000:0.0:1787829738.342159:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829738.342164:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829738.342175:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829738.342184:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280924160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829738.342193:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280924288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829738.342207:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829738.342212:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829738.342214:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829738.342219:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4287 00000100:00080000:0.0:1787829738.342223:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829738.342229:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829738.342235:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829738.342248:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280924416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829738.342263:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829738.342263:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829738.342269:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829738.342270:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829738.342283:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4287 00000100:00080000:0.0:1787829738.342289:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829738.342294:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829738.342301:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829738.342316:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280924544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829738.342328:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829738.342328:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829738/real 1787829738] req@ffff8ab3f075c000 x1874671280924160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829738.342338:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829738.342342:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829738.342348:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4287 00000100:00080000:0.0:1787829738.342352:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829738.342356:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280924160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829738.342360:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829738.342369:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787829738.342373:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829738.342379:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787829738.342382:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280924672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829738.342386:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280924160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829738.342417:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280924288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829738.342437:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829738.342447:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829738/real 1787829738] req@ffff8ab3f075cf00 x1874671280924288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829738.342455:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280924288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829738.342461:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829738.342463:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829738.342465:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280924288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829738.342469:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280924416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829738.342479:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829738.342507:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829738/real 1787829738] req@ffff8ab3f075de00 x1874671280924416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829738.342515:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280924416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829738.342521:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829738.342523:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829738.342525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280924416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829738.342529:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280924544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829738.342544:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829738.342549:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829738/real 1787829738] req@ffff8ab3f075f0c0 x1874671280924544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829738.342557:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280924544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829738.342564:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829738.342566:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829738.342567:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280924544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829738.342571:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280924672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829738.342581:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829738.342586:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829738/real 1787829738] req@ffff8ab3f075cb40 x1874671280924672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829738.342593:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280924672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829738.342600:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829738.342602:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829738.342604:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280924672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829739.366071:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829740.390060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829741.414075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829742.438051:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829743.462020:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829743.462031:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829743.462036:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829743.462047:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4293 00000100:00080000:2.0:1787829743.462058:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829743.462066:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829743.462072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829743.462112:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280924800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829743.462136:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829743.462139:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829743.462141:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829743.462143:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4293 00000100:00080000:2.0:1787829743.462147:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829743.462150:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829743.462161:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829743.462173:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280924928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829743.462178:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280924800:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829743.462185:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829743.462188:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829743.462190:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829743.462194:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4293 00000100:00080000:2.0:1787829743.462196:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829743.462199:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829743.462203:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829743.462212:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280925056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829743.462218:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829743.462221:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829743.462222:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829743.462226:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4293 00000100:00080000:2.0:1787829743.462237:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829743.462240:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829743.462244:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829743.462249:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829743.462256:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280925184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829743.462264:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829743.462267:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829743.462268:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829743.462271:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4293 00000100:00080000:2.0:1787829743.462280:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829743.462283:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829743.462286:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829743.462294:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280925312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829743.462312:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829743/real 1787829743] req@ffff8ab3f043b840 x1874671280924800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829743.462331:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280924800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829743.462341:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829743.462344:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829743.462347:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280924800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829743.462370:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280924928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829743.462387:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829743.462396:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829743/real 1787829743] req@ffff8ab3f043b480 x1874671280924928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829743.462404:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280924928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829743.462410:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829743.462413:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829743.462415:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280924928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829743.462420:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280925056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829743.462430:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829743.462436:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829743/real 1787829743] req@ffff8ab3f0439e00 x1874671280925056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829743.462444:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280925056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829743.462465:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829743.462468:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829743.462470:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280925056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829743.462474:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280925184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829743.462490:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829743.462503:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829743/real 1787829743] req@ffff8ab3f043a1c0 x1874671280925184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829743.462511:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280925184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829743.462518:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829743.462520:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829743.462522:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280925184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829743.462526:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280925312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829743.462537:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829743.462543:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829743/real 1787829743] req@ffff8ab3f043a940 x1874671280925312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829743.462551:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280925312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829743.462566:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829743.462568:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829743.462570:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280925312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829744.486025:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829745.510058:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829746.533989:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829747.558042:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829748.581988:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829748.581999:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829748.582003:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829748.582015:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4298 00000100:00080000:0.0:1787829748.582026:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829748.582035:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829748.582065:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829748.582082:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280925440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829748.582108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829748.582111:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829748.582112:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829748.582116:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4298 00000100:00080000:0.0:1787829748.582119:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829748.582122:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829748.582138:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829748.582145:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280925440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829748.582154:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280925568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829748.582171:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829748.582174:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829748.582177:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829748.582181:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4298 00000100:00080000:0.0:1787829748.582184:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829748.582187:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829748.582191:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829748.582206:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280925696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829748.582215:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829748.582218:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829748.582220:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829748.582225:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4298 00000100:00080000:0.0:1787829748.582228:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829748.582232:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829748.582239:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829748.582263:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280925824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829748.582276:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829748.582282:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280925568:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787829748.582284:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829748.582292:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829748.582302:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4298 00000100:00080000:0.0:1787829748.582316:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829748.582320:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280925696:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787829748.582326:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829748.582336:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829748.582350:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280925824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787829748.582353:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280925952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829748.582396:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280925952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829749.606085:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829750.630100:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829751.654095:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787829751.654212:0:10942:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787829751.654260:0:10942:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829751.654279:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829751.654319:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829751.654331:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829751.654340:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829751.654349:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787829751.654352:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829748/real 1787829751] req@ffff8ab3f075e940 x1874671280925568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829751.654382:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280925568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829751.654427:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829751.654433:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829751.654437:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280925568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829751.654447:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829748/real 1787829751] req@ffff8ab3f075e580 x1874671280925696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829751.654468:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280925696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829751.654477:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829751.654481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829751.654485:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280925696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829751.654493:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829748/real 1787829751] req@ffff8ab3f075ed00 x1874671280925824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829751.654506:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280925824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829751.654517:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829751.654521:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829751.654525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280925824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829751.654532:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829748/real 1787829751] req@ffff8ab3f075da40 x1874671280925440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829751.654545:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280925440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829751.654553:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829751.654556:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829751.654559:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280925440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829751.654589:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829748/real 1787829751] req@ffff8ab3f075f840 x1874671280925952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829751.654604:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280925952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829751.654614:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829751.654618:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829751.654621:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280925952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829752.678088:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829753.702050:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829753.702061:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1787829753.702063:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829753.702066:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829753.702080:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4303 00000100:00080000:0.0:1787829753.702103:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829753.702112:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829753.702159:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280926080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829753.702191:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829753.702195:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829753.702197:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829753.702201:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4303 00000100:00080000:0.0:1787829753.702205:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829753.702209:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829753.702221:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829753.702237:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280926208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829753.702242:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280926080:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829753.702251:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829753.702255:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829753.702257:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829753.702262:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4303 00000100:00080000:0.0:1787829753.702271:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829753.702275:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829753.702281:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829753.702310:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280926336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829753.702328:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829753.702333:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829753.702334:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829753.702339:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829753.702345:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4303 00000100:00080000:0.0:1787829753.702349:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829753.702355:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829753.702361:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829753.702387:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280926464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829753.702397:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829753.702401:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829753.702403:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829753.702408:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4303 00000100:00080000:0.0:1787829753.702413:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829753.702419:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829753/real 1787829753] req@ffff8ab3f075f480 x1874671280926080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829753.702420:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829753.702427:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829753.702445:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280926592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829753.702451:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280926080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829753.702472:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829753.702477:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829753.702483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280926080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829753.702516:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280926208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829753.702543:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829753.702565:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829753/real 1787829753] req@ffff8ab3f075f840 x1874671280926208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829753.702579:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280926208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829753.702593:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829753.702597:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829753.702601:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280926208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829753.702607:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280926336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829753.702629:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829753.702642:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829753/real 1787829753] req@ffff8ab3f075da40 x1874671280926336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829753.702655:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280926336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829753.702664:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829753.702667:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829753.702671:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280926336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829753.702690:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280926464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829753.702716:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829753.702728:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829753/real 1787829753] req@ffff8ab3f075ed00 x1874671280926464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829753.702740:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280926464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829753.702751:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829753.702755:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829753.702759:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280926464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829753.702766:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280926592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829753.702786:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829753.702798:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829753/real 1787829753] req@ffff8ab3f075e580 x1874671280926592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829753.702811:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280926592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829753.702822:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829753.702825:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829753.702829:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280926592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829754.726115:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829755.750070:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829756.774101:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829757.798064:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829758.821993:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829758.822006:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829758.822012:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829758.822023:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4308 00000100:00080000:2.0:1787829758.822037:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829758.822046:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829758.822055:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829758.822097:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280926720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829758.822122:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829758.822126:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829758.822128:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829758.822132:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4308 00000100:00080000:2.0:1787829758.822137:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829758.822142:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829758.822155:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829758.822160:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280926720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829758.822168:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280926848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829758.822178:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829758.822181:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829758.822184:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829758.822188:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4308 00000100:00080000:2.0:1787829758.822191:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829758.822195:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829758.822199:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829758.822208:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280926976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829758.822217:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829758.822220:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829758.822221:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829758.822226:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4308 00000100:00080000:2.0:1787829758.822228:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829758.822233:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829758.822233:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829758.822237:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829758.822247:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280927104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829758.822255:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829758.822258:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829758.822260:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829758.822263:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4308 00000100:00080000:2.0:1787829758.822267:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829758.822271:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829758.822275:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829758.822288:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829758/real 1787829758] req@ffff8ab3f0439a40 x1874671280926720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829758.822288:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280927232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829758.822308:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280926720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829758.822319:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829758.822324:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829758.822328:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280926720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829758.822367:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280926848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829758.822386:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829758.822397:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829758/real 1787829758] req@ffff8ab3f043bc00 x1874671280926848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829758.822405:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280926848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829758.822411:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829758.822413:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829758.822416:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280926848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829758.822419:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280926976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829758.822429:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829758.822434:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829758/real 1787829758] req@ffff8ab3f04383c0 x1874671280926976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829758.822441:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280926976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829758.822446:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829758.822448:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829758.822450:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280926976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829758.822453:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280927104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829758.822464:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829758.822469:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829758/real 1787829758] req@ffff8ab3f043ad00 x1874671280927104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829758.822476:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280927104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829758.822484:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829758.822486:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829758.822488:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280927104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829758.822491:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280927232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829758.822500:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829758.822505:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829758/real 1787829758] req@ffff8ab3f04392c0 x1874671280927232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829758.822512:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280927232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829758.822519:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829758.822521:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829758.822523:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280927232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829759.846062:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829760.870139:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829761.894142:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829762.918077:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829763.942011:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829763.942023:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829763.942027:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829763.942039:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4313 00000100:00080000:0.0:1787829763.942054:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829763.942066:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829763.942077:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829763.942119:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280927360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829763.942145:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829763.942148:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829763.942150:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829763.942153:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4313 00000100:00080000:0.0:1787829763.942156:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829763.942159:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829763.942170:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829763.942183:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280927488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829763.942195:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829763.942201:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829763.942205:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829763.942205:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280927360:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829763.942214:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4313 00000100:00080000:0.0:1787829763.942223:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829763.942230:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829763.942233:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829763.942243:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280927616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829763.942250:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829763.942253:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829763.942255:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829763.942258:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4313 00000100:00080000:0.0:1787829763.942279:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829763.942283:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829763.942287:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829763.942295:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829763.942314:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280927744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829763.942325:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829763.942329:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829763.942331:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829763.942335:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4313 00000100:00080000:0.0:1787829763.942339:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829763.942343:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829763.942349:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829763.942363:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280927872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829763.942373:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829763/real 1787829763] req@ffff8ab3f075e940 x1874671280927360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829763.942395:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280927360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829763.942407:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829763.942412:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829763.942416:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280927360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829763.942441:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280927488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829763.942465:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829763.942478:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829763/real 1787829763] req@ffff8ab3f075cb40 x1874671280927488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829763.942490:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280927488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829763.942499:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829763.942503:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829763.942506:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280927488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829763.942513:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280927616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829763.942530:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829763.942541:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829763/real 1787829763] req@ffff8ab3f075f0c0 x1874671280927616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829763.942552:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280927616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829763.942560:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829763.942564:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829763.942567:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280927616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829763.942573:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280927744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829763.942590:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829763.942600:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829763/real 1787829763] req@ffff8ab3f075de00 x1874671280927744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829763.942612:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280927744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829763.942622:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829763.942625:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829763.942629:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280927744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829763.942635:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280927872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829763.942652:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829763.942662:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829763/real 1787829763] req@ffff8ab3f075cf00 x1874671280927872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829763.942674:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280927872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829763.942684:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829763.942687:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829763.942690:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280927872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829764.966091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829765.990067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829767.014106:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829768.038037:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829769.062039:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829769.062050:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829769.062054:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829769.062065:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4318 00000100:00080000:0.0:1787829769.062076:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829769.062084:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829769.062088:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829769.062122:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280928000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829769.062143:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829769.062145:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829769.062147:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829769.062149:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4318 00000100:00080000:0.0:1787829769.062152:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829769.062155:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829769.062163:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829769.062174:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280928128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829769.062182:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829769.062183:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280928000:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829769.062188:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829769.062191:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829769.062195:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4318 00000100:00080000:0.0:1787829769.062197:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829769.062199:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829769.062203:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829769.062212:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280928256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829769.062219:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829769.062222:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829769.062223:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829769.062225:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4318 00000100:00080000:0.0:1787829769.062235:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829769.062239:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829769.062242:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829769.062244:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829769.062256:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280928384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829769.062264:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829769.062281:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829769.062284:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829769.062288:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4318 00000100:00080000:0.0:1787829769.062292:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829769.062295:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829769.062299:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829769.062307:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829769/real 1787829769] req@ffff8ab3f075c000 x1874671280928000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829769.062311:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280928512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829769.062352:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280928000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829769.062362:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829769.062365:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829769.062368:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280928000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829769.062386:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280928128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829769.062404:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829769.062413:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829769/real 1787829769] req@ffff8ab3f075cf00 x1874671280928128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829769.062422:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280928128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829769.062427:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829769.062430:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829769.062432:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280928128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829769.062436:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280928256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829769.062446:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829769.062451:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829769/real 1787829769] req@ffff8ab3f075de00 x1874671280928256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829769.062459:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280928256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829769.062464:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829769.062466:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829769.062468:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280928256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829769.062472:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280928384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829769.062482:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829769.062487:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829769/real 1787829769] req@ffff8ab3f075f0c0 x1874671280928384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829769.062494:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280928384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829769.062500:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829769.062502:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829769.062504:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280928384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829769.062507:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280928512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829769.062516:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829769.062524:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829769/real 1787829769] req@ffff8ab3f075cb40 x1874671280928512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829769.062535:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280928512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829769.062546:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829769.062550:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829769.062553:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280928512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829770.086127:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829771.110094:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829772.134087:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829773.158096:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829774.182038:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829774.182053:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829774.182059:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1787829774.182072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829774.182072:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4323 00000100:00080000:2.0:1787829774.182088:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829774.182097:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829774.182158:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280928640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829774.182186:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829774.182191:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829774.182193:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829774.182196:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4323 00000100:00080000:2.0:1787829774.182200:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829774.182206:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829774.182218:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829774.182246:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280928640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829774.182246:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280928768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829774.182265:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829774.182269:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829774.182272:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829774.182276:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4323 00000100:00080000:2.0:1787829774.182280:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829774.182285:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829774.182291:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829774.182304:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280928896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829774.182314:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829774.182319:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829774.182321:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829774.182325:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4323 00000100:00080000:2.0:1787829774.182329:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829774.182335:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829774.182337:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829774.182342:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829774.182355:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280929024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829774.182367:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829774.182371:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829774.182372:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829774.182377:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4323 00000100:00080000:2.0:1787829774.182381:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829774.182394:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829774.182399:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829774.182402:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829774/real 1787829774] req@ffff8ab3f0439680 x1874671280928640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829774.182416:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280929152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829774.182431:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280928640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829774.182445:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829774.182451:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829774.182456:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280928640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829774.182486:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280928768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829774.182512:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829774.182528:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829774/real 1787829774] req@ffff8ab3f043b0c0 x1874671280928768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829774.182542:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280928768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829774.182551:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829774.182555:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829774.182558:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280928768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829774.182565:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280928896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829774.182585:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829774.182598:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829774/real 1787829774] req@ffff8ab3f043a580 x1874671280928896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829774.182611:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280928896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829774.182626:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829774.182629:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829774.182633:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280928896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829774.182639:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280929024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829774.182659:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829774.182671:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829774/real 1787829774] req@ffff8ab3f0438f00 x1874671280929024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829774.182684:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280929024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829774.182694:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829774.182697:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829774.182701:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280929024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829774.182707:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280929152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829774.182725:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829774.182736:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829774/real 1787829774] req@ffff8ab3f0438b40 x1874671280929152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829774.182747:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280929152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829774.182757:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829774.182761:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829774.182765:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280929152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829775.206079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829776.230097:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829777.254079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829778.278055:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829779.302012:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829779.302023:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829779.302027:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829779.302038:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4328 00000100:00080000:0.0:1787829779.302048:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787829779.302054:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829779.302055:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829779.302096:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280929280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829779.302117:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829779.302120:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829779.302121:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829779.302123:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4328 00000100:00080000:0.0:1787829779.302126:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829779.302128:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829779.302138:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829779.302148:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280929408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829779.302150:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280929280:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829779.302163:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829779.302165:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829779.302166:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829779.302169:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4328 00000100:00080000:0.0:1787829779.302171:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829779.302174:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829779.302177:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829779.302186:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280929536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829779.302192:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829779.302209:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829779.302211:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829779.302215:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4328 00000100:00080000:0.0:1787829779.302226:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829779.302230:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829779.302234:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829779.302238:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829779.302245:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280929664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829779.302252:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829779.302255:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829779.302257:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829779.302260:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4328 00000100:00080000:0.0:1787829779.302263:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829779.302265:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829779.302267:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829779.302276:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280929792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829779.302293:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829779/real 1787829779] req@ffff8ab3f075e940 x1874671280929280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829779.302307:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280929280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829779.302315:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829779.302318:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829779.302321:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280929280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829779.302341:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280929408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829779.302356:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829779.302364:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829779/real 1787829779] req@ffff8ab3f075e580 x1874671280929408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829779.302370:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280929408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829779.302375:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829779.302377:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829779.302379:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280929408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829779.302382:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280929536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829779.302390:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829779.302396:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829779/real 1787829779] req@ffff8ab3f075ed00 x1874671280929536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829779.302402:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280929536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829779.302406:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829779.302408:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829779.302410:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280929536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829779.302413:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280929664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829779.302422:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829779.302426:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829779/real 1787829779] req@ffff8ab3f075da40 x1874671280929664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829779.302433:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280929664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829779.302438:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829779.302440:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829779.302441:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280929664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829779.302444:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280929792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829779.302452:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829779.302457:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829779/real 1787829779] req@ffff8ab3f075f840 x1874671280929792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829779.302462:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280929792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829779.302468:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829779.302470:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829779.302472:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280929792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829780.326062:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829781.350081:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829782.374046:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829783.398033:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829784.422020:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829784.422033:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829784.422038:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829784.422049:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4334 00000100:00080000:2.0:1787829784.422061:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829784.422069:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829784.422115:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280929920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787829784.422121:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829784.422151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829784.422156:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829784.422157:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829784.422174:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4334 00000100:00080000:2.0:1787829784.422178:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829784.422183:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829784.422195:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829784.422208:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280929920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829784.422211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280930048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829784.422225:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829784.422229:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829784.422231:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829784.422235:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4334 00000100:00080000:2.0:1787829784.422238:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829784.422242:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829784.422247:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829784.422258:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280930176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829784.422268:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829784.422272:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829784.422274:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829784.422280:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4334 00000800:00000100:1.0:1787829784.422281:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829784.422284:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829784.422289:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829784.422294:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829784.422306:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280930304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829784.422316:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829784.422320:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829784.422322:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829784.422327:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4334 00000100:00080000:2.0:1787829784.422330:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829784.422335:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829784.422341:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829784.422342:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829784/real 1787829784] req@ffff8ab3f0438780 x1874671280929920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829784.422356:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280930432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829784.422371:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280929920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829784.422383:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829784.422388:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829784.422392:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280929920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829784.422417:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280930048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829784.422437:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829784.422450:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829784/real 1787829784] req@ffff8ab3f0438b40 x1874671280930048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829784.422462:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280930048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829784.422470:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829784.422477:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829784.422481:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280930048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829784.422486:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280930176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829784.422501:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829784.422511:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829784/real 1787829784] req@ffff8ab3f0438f00 x1874671280930176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829784.422522:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280930176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829784.422535:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829784.422538:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829784.422542:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280930176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829784.422548:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280930304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829784.422564:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829784.422571:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829784/real 1787829784] req@ffff8ab3f043a580 x1874671280930304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829784.422578:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280930304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829784.422585:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829784.422593:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829784.422595:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280930304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829784.422598:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280930432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829784.422608:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829784.422614:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829784/real 1787829784] req@ffff8ab3f043b0c0 x1874671280930432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829784.422620:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280930432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829784.422626:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829784.422628:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829784.422629:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280930432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829785.446048:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829786.470091:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829787.494053:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829788.518125:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829789.542036:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829789.542048:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829789.542052:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829789.542066:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4339 00000100:00080000:2.0:1787829789.542078:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829789.542087:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829789.542114:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829789.542143:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280930560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829789.542174:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829789.542178:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829789.542180:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829789.542185:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4339 00000100:00080000:2.0:1787829789.542189:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829789.542195:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829789.542206:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829789.542226:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280930560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829789.542226:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280930688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829789.542251:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829789.542254:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829789.542256:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829789.542260:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4339 00000100:00080000:2.0:1787829789.542263:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829789.542267:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829789.542271:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829789.542284:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280930816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829789.542291:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829789.542294:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829789.542295:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829789.542299:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4339 00000100:00080000:2.0:1787829789.542312:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829789.542315:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829789.542319:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829789.542326:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829789.542334:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280930944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829789.542357:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829789.542361:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829789.542363:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829789.542370:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4339 00000100:00080000:2.0:1787829789.542376:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829789.542382:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829789.542390:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829789.542391:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829789/real 1787829789] req@ffff8ab3f0439680 x1874671280930560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829789.542416:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280931072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829789.542422:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280930560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829789.542459:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829789.542475:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829789.542481:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280930560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829789.542510:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280930688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829789.542535:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829789.542549:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829789/real 1787829789] req@ffff8ab3f04392c0 x1874671280930688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829789.542563:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280930688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829789.542580:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829789.542583:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829789.542587:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280930688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829789.542594:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280930816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829789.542614:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829789.542627:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829789/real 1787829789] req@ffff8ab3f043ad00 x1874671280930816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829789.542639:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280930816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829789.542647:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829789.542651:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829789.542654:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280930816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829789.542661:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280930944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829789.542680:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829789.542691:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829789/real 1787829789] req@ffff8ab3f04383c0 x1874671280930944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829789.542704:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280930944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829789.542714:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829789.542717:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829789.542721:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280930944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829789.542727:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280931072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829789.542746:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829789.542758:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829789/real 1787829789] req@ffff8ab3f043bc00 x1874671280931072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829789.542770:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280931072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829789.542780:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829789.542783:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829789.542786:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280931072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829790.566185:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829791.590047:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829792.614066:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829793.638049:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829794.662000:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829794.662012:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829794.662016:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829794.662027:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4344 00000100:00080000:2.0:1787829794.662038:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829794.662047:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829794.662084:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829794.662108:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280931200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829794.662133:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829794.662135:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829794.662137:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829794.662140:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4344 00000100:00080000:2.0:1787829794.662142:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829794.662146:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829794.662156:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829794.662168:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280931328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829794.662173:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280931200:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829794.662179:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829794.662183:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829794.662184:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829794.662188:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4344 00000100:00080000:2.0:1787829794.662190:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829794.662194:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829794.662198:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829794.662209:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280931456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829794.662216:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829794.662219:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829794.662220:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829794.662223:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4344 00000100:00080000:2.0:1787829794.662235:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829794.662239:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829794.662242:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829794.662247:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829794.662252:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280931584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829794.662259:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829794.662262:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829794.662263:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829794.662266:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4344 00000100:00080000:2.0:1787829794.662271:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829794.662273:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829794.662276:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829794.662284:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280931712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829794.662303:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829794/real 1787829794] req@ffff8ab3f0439a40 x1874671280931200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829794.662320:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280931200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829794.662329:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829794.662332:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829794.662335:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280931200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829794.662357:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280931328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829794.662371:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829794.662380:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829794/real 1787829794] req@ffff8ab3f043a940 x1874671280931328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829794.662388:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280931328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829794.662394:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829794.662396:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829794.662398:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280931328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829794.662402:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280931456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829794.662412:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829794.662417:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829794/real 1787829794] req@ffff8ab3f043a1c0 x1874671280931456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829794.662424:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280931456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829794.662429:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829794.662431:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829794.662433:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280931456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829794.662436:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280931584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829794.662447:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829794.662452:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829794/real 1787829794] req@ffff8ab3f0439e00 x1874671280931584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829794.662459:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280931584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829794.662467:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829794.662469:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829794.662471:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280931584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829794.662474:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280931712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829794.662483:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829794.662488:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829794/real 1787829794] req@ffff8ab3f043b480 x1874671280931712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829794.662495:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280931712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829794.662501:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829794.662503:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829794.662505:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280931712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829795.686180:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829796.710094:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829797.734099:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829798.758136:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829799.781990:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829799.782002:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829799.782006:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829799.782017:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4349 00000100:00080000:2.0:1787829799.782028:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829799.782036:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829799.782052:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829799.782096:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280931840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829799.782122:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829799.782125:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829799.782127:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829799.782130:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4349 00000100:00080000:2.0:1787829799.782133:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829799.782137:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829799.782150:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829799.782162:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280931840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829799.782170:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280931968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829799.782184:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829799.782188:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829799.782191:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829799.782195:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4349 00000100:00080000:2.0:1787829799.782199:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829799.782203:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829799.782208:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829799.782219:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280932096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829799.782227:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829799.782230:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829799.782234:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829799.782235:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829799.782241:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4349 00000100:00080000:2.0:1787829799.782246:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829799.782250:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829799.782255:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829799.782263:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280932224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829799.782270:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829799.782273:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829799.782274:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829799.782278:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4349 00000100:00080000:2.0:1787829799.782281:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829799.782284:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829799.782288:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829799/real 1787829799] req@ffff8ab3f043b840 x1874671280931840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829799.782288:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829799.782300:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280932352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829799.782309:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280931840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829799.782320:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829799.782331:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829799.782334:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280931840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829799.782358:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280931968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829799.782374:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829799.782383:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829799/real 1787829799] req@ffff8ab3f043b480 x1874671280931968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829799.782392:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280931968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829799.782398:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829799.782400:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829799.782403:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280931968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829799.782406:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280932096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829799.782416:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829799.782422:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829799/real 1787829799] req@ffff8ab3f0439e00 x1874671280932096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829799.782429:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280932096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829799.782435:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829799.782436:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829799.782438:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280932096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829799.782442:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280932224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829799.782452:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829799.782458:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829799/real 1787829799] req@ffff8ab3f043a1c0 x1874671280932224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829799.782466:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280932224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829799.782473:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829799.782475:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829799.782476:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280932224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829799.782480:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280932352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829799.782489:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829799.782495:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829799/real 1787829799] req@ffff8ab3f043a940 x1874671280932352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829799.782502:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280932352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829799.782509:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829799.782511:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829799.782512:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280932352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829800.806032:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829801.830108:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829802.854086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829803.878081:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829804.901977:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829804.901991:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829804.901996:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829804.902009:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4354 00000100:00080000:2.0:1787829804.902035:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829804.902043:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829804.902060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829804.902090:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280932480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829804.902115:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829804.902118:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829804.902119:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829804.902122:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4354 00000100:00080000:2.0:1787829804.902125:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829804.902128:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829804.902138:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829804.902148:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280932608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829804.902156:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829804.902159:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280932480:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829804.902159:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829804.902164:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829804.902170:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4354 00000100:00080000:2.0:1787829804.902172:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829804.902175:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829804.902179:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829804.902195:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280932736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829804.902207:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829804.902212:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829804.902214:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829804.902220:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4354 00000100:00080000:2.0:1787829804.902232:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829804.902238:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829804.902244:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829804.902247:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829804.902264:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280932864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829804.902275:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829804.902278:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829804.902281:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829804.902294:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829804/real 1787829804] req@ffff8ab3f0439a40 x1874671280932480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829804.902295:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4354 00000100:00080000:2.0:1787829804.902301:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829804.902307:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829804.902314:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829804.902333:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280932992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829804.902334:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280932480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829804.902355:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829804.902360:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829804.902364:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280932480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829804.902388:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280932608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829804.902407:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829804.902418:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829804/real 1787829804] req@ffff8ab3f043bc00 x1874671280932608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829804.902426:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280932608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829804.902432:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829804.902434:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829804.902437:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280932608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829804.902440:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280932736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829804.902450:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829804.902456:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829804/real 1787829804] req@ffff8ab3f04383c0 x1874671280932736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829804.902463:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280932736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829804.902468:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829804.902470:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829804.902473:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280932736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829804.902476:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280932864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829804.902487:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829804.902492:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829804/real 1787829804] req@ffff8ab3f043ad00 x1874671280932864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829804.902500:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280932864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829804.902506:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829804.902508:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829804.902510:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280932864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829804.902513:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280932992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829804.902522:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829804.902530:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829804/real 1787829804] req@ffff8ab3f04392c0 x1874671280932992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829804.902542:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280932992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829804.902553:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829804.902556:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829804.902559:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280932992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829805.926098:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829806.950092:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829807.974073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829808.998072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829810.021965:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829810.021974:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829810.021978:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829810.022003:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4359 00000100:00080000:2.0:1787829810.022013:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829810.022019:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829810.022031:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829810.022053:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280933120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829810.022075:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829810.022078:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829810.022079:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829810.022082:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4359 00000100:00080000:2.0:1787829810.022084:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829810.022088:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829810.022096:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829810.022103:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280933120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829810.022106:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280933248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829810.022117:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829810.022119:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829810.022121:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829810.022123:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4359 00000100:00080000:2.0:1787829810.022125:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829810.022128:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829810.022131:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829810.022141:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280933376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829810.022147:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829810.022149:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829810.022151:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829810.022154:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4359 00000100:00080000:2.0:1787829810.022156:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829810.022162:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829810.022164:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829810.022167:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829810.022177:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280933504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829810.022184:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829810.022186:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829810.022187:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829810.022190:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4359 00000100:00080000:2.0:1787829810.022192:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829810.022195:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829810.022198:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829810.022210:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829810/real 1787829810] req@ffff8ab3f0439680 x1874671280933120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829810.022210:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280933632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829810.022240:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280933120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829810.022249:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829810.022253:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829810.022256:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280933120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829810.022274:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280933248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829810.022288:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829810.022296:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829810/real 1787829810] req@ffff8ab3f043b0c0 x1874671280933248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829810.022304:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280933248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829810.022310:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829810.022312:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829810.022314:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280933248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829810.022319:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280933376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829810.022329:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829810.022335:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829810/real 1787829810] req@ffff8ab3f043a580 x1874671280933376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829810.022347:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280933376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829810.022362:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829810.022366:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829810.022370:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280933376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829810.022377:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280933504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829810.022398:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829810.022407:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829810/real 1787829810] req@ffff8ab3f0438f00 x1874671280933504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829810.022416:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280933504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829810.022423:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829810.022433:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829810.022436:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280933504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829810.022440:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280933632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829810.022453:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829810.022459:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829810/real 1787829810] req@ffff8ab3f0438b40 x1874671280933632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829810.022467:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280933632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829810.022473:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829810.022475:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829810.022478:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280933632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829811.046130:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829812.070107:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829813.094067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829814.118068:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829815.142026:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829815.142034:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829815.142038:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829815.142047:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4364 00000100:00080000:0.0:1787829815.142056:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829815.142062:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829815.142091:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829815.142095:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280933760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829815.142115:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829815.142118:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829815.142119:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829815.142122:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4364 00000100:00080000:0.0:1787829815.142124:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829815.142127:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829815.142134:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829815.142145:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280933888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829815.142153:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829815.142156:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280933760:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829815.142157:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829815.142160:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829815.142167:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4364 00000100:00080000:0.0:1787829815.142170:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829815.142173:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829815.142176:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829815.142185:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280934016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829815.142191:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829815.142193:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829815.142195:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829815.142198:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4364 00000100:00080000:0.0:1787829815.142208:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829815.142211:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829815.142215:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829815.142223:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280934144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829815.142229:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829815.142232:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829815.142234:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829815.142238:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4364 00000100:00080000:0.0:1787829815.142240:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829815.142244:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829815.142248:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829815.142256:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280934272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829815.142261:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280933888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829815.142289:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280934016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829815.142313:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280934144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829815.142328:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280934272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829816.167013:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829817.190126:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829818.214102:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829818.214285:0:10944:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787829818.214350:0:10944:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829818.214371:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829818.214427:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829818.214438:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829818.214453:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829818.214464:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787829818.214681:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829815/real 1787829818] req@ffff8ab3f075e580 x1874671280934272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829818.214714:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280934272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829818.214732:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829818.214740:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829818.214745:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280934272:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829818.214754:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829815/real 1787829818] req@ffff8ab3f075f840 x1874671280933888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829818.214769:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280933888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829818.214779:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829818.214783:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829818.214787:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280933888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829818.214795:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829815/real 1787829818] req@ffff8ab3f075da40 x1874671280934016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829818.214809:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280934016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829818.214818:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829818.214822:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829818.214826:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280934016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829818.214834:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829815/real 1787829818] req@ffff8ab3f075ed00 x1874671280934144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829818.214847:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280934144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829818.214858:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829818.214862:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829818.214865:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280934144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829818.214872:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829815/real 1787829818] req@ffff8ab3f075f480 x1874671280933760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829818.214903:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280933760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829818.214913:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829818.214916:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829818.214920:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280933760:192.168.203.124@tcp:250:kworker.0 00010000:00010000:3.0:1787829819.238071:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829820.262037:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1787829820.262060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829820.262068:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829820.262074:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829820.262087:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4369 00000100:00080000:2.0:1787829820.262100:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829820.262108:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829820.262165:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280934400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829820.262215:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829820.262223:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829820.262225:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829820.262229:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4369 00000100:00080000:2.0:1787829820.262233:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829820.262239:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829820.262258:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829820.262282:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280934528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829820.262293:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829820.262297:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829820.262299:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829820.262304:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4369 00000100:00080000:2.0:1787829820.262308:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829820.262312:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829820.262317:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829820.262337:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280934656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829820.262349:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829820.262353:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829820.262361:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829820.262366:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4369 00000100:00080000:2.0:1787829820.262371:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829820.262378:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280934400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829820.262380:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829820.262401:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829820.262418:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280934784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829820.262438:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829820.262443:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829820.262445:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829820.262452:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4369 00000100:00080000:2.0:1787829820.262460:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829820.262465:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829820.262476:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829820.262479:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829820.262495:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280934912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829820.262548:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829820/real 1787829820] req@ffff8ab3f0438780 x1874671280934400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829820.262573:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280934400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829820.262586:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829820.262591:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829820.262596:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280934400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829820.262606:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280934528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829820.262633:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829820.262648:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829820/real 1787829820] req@ffff8ab3f0438b40 x1874671280934528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829820.262662:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280934528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829820.262672:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829820.262676:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829820.262679:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280934528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829820.262686:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280934656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829820.262707:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829820.262735:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829820/real 1787829820] req@ffff8ab3f0438f00 x1874671280934656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829820.262749:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280934656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829820.262758:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829820.262762:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829820.262766:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280934656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829820.262799:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280934784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829820.262822:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829820.262834:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829820/real 1787829820] req@ffff8ab3f043a580 x1874671280934784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829820.262847:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280934784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829820.262858:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829820.262862:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829820.262866:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280934784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829820.262872:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280934912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829820.262934:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829820.262948:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829820/real 1787829820] req@ffff8ab3f043b0c0 x1874671280934912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829820.262961:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280934912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829820.262978:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829820.262982:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829820.262985:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280934912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829821.286085:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829822.310039:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829823.334064:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829824.358072:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829825.381989:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829825.382001:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829825.382006:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829825.382018:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4375 00000100:00080000:2.0:1787829825.382027:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829825.382034:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829825.382066:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829825.382076:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280935040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829825.382099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829825.382101:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829825.382103:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829825.382105:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4375 00000100:00080000:2.0:1787829825.382109:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829825.382114:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829825.382125:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829825.382142:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280935040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829825.382144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280935168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829825.382167:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829825.382171:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829825.382173:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829825.382178:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4375 00000100:00080000:2.0:1787829825.382181:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829825.382185:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829825.382190:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829825.382202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280935296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829825.382211:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829825.382212:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829825.382218:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829825.382220:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829825.382225:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4375 00000100:00080000:2.0:1787829825.382229:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829825.382232:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829825.382236:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829825.382243:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280935424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829825.382255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829825.382257:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829825.382259:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829825.382263:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4375 00000100:00100000:1.0:1787829825.382266:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829825/real 1787829825] req@ffff8ab3f0439680 x1874671280935040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829825.382266:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829825.382271:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829825.382279:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829825.382293:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280935552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829825.382305:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280935040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829825.382319:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829825.382323:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829825.382328:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280935040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829825.382351:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280935168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829825.382374:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829825.382386:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829825/real 1787829825] req@ffff8ab3f04392c0 x1874671280935168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829825.382398:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280935168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829825.382407:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829825.382410:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829825.382413:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280935168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829825.382420:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280935296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829825.382439:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829825.382449:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829825/real 1787829825] req@ffff8ab3f043ad00 x1874671280935296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829825.382459:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280935296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829825.382466:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829825.382469:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829825.382473:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280935296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829825.382478:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280935424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829825.382496:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829825.382506:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829825/real 1787829825] req@ffff8ab3f04383c0 x1874671280935424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829825.382517:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280935424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829825.382526:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829825.382529:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829825.382531:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280935424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829825.382537:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280935552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829825.382554:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829825.382564:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829825/real 1787829825] req@ffff8ab3f043bc00 x1874671280935552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829825.382573:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280935552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829825.382582:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829825.382598:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829825.382602:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280935552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829826.406050:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829827.430094:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829828.454078:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829829.478073:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829830.501996:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829830.502006:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829830.502010:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829830.502020:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4380 00000100:00080000:2.0:1787829830.502029:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829830.502037:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829830.502048:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829830.502075:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280935680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829830.502099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829830.502103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829830.502105:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829830.502110:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4380 00000100:00080000:2.0:1787829830.502114:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829830.502119:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829830.502122:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280935680:192.168.203.124@tcp:250:kworker.0 00000008:00100000:2.0:1787829830.502130:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829830.502144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280935808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829830.502151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829830.502154:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829830.502155:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829830.502158:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4380 00000100:00080000:2.0:1787829830.502167:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829830.502170:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829830.502173:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829830.502176:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829830.502186:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280935936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829830.502192:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829830.502194:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829830.502195:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829830.502197:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4380 00000100:00080000:2.0:1787829830.502199:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829830.502201:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829830.502205:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829830.502213:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280936064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829830.502219:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829830.502221:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829830/real 1787829830] req@ffff8ab3f0439a40 x1874671280935680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829830.502221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829830.502224:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829830.502231:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4380 00000100:00080000:2.0:1787829830.502236:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829830.502239:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829830.502242:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829830.502247:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280935680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829830.502250:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280936192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829830.502263:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829830.502268:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829830.502271:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280935680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829830.502289:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280935808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829830.502322:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829830.502334:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829830/real 1787829830] req@ffff8ab3f043a940 x1874671280935808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829830.502343:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280935808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829830.502352:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829830.502354:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829830.502356:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280935808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829830.502360:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280935936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829830.502372:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829830.502377:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829830/real 1787829830] req@ffff8ab3f043a1c0 x1874671280935936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829830.502384:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280935936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829830.502388:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829830.502390:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829830.502392:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280935936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829830.502395:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280936064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829830.502403:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829830.502408:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829830/real 1787829830] req@ffff8ab3f0439e00 x1874671280936064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829830.502414:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280936064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829830.502420:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829830.502421:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829830.502423:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280936064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829830.502426:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280936192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829830.502434:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829830.502438:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829830/real 1787829830] req@ffff8ab3f043b480 x1874671280936192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829830.502444:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280936192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829830.502449:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829830.502450:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829830.502452:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280936192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829831.526082:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829832.550089:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829833.574067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829834.598019:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829835.621978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829835.621986:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829835.621989:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829835.621998:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4385 00000100:00080000:2.0:1787829835.622005:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829835.622011:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829835.622039:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280936320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787829835.622046:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829835.622056:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829835.622060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829835.622061:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829835.622064:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4385 00000100:00080000:2.0:1787829835.622066:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829835.622073:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829835.622080:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829835.622088:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280936320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829835.622093:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280936448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829835.622105:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829835.622107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829835.622109:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829835.622111:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4385 00000100:00080000:2.0:1787829835.622117:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829835.622120:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829835.622123:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829835.622130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280936576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829835.622136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829835.622138:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829835.622139:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829835.622141:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4385 00000800:00000100:1.0:1787829835.622145:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829835.622146:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829835.622153:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829835.622159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829835.622170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280936704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829835.622180:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829835.622184:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829835.622186:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829835.622188:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829835/real 1787829835] req@ffff8ab3f043b840 x1874671280936320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829835.622193:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4385 00000100:00080000:2.0:1787829835.622198:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829835.622204:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829835.622209:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829835.622214:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280936320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829835.622219:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280936832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829835.622236:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829835.622241:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829835.622247:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280936320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829835.622266:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280936448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829835.622284:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829835.622295:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829835/real 1787829835] req@ffff8ab3f043b480 x1874671280936448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829835.622307:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280936448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829835.622315:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829835.622317:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829835.622321:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280936448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829835.622326:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280936576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829835.622337:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829835.622345:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829835/real 1787829835] req@ffff8ab3f0439e00 x1874671280936576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829835.622356:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280936576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829835.622363:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829835.622366:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829835.622368:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280936576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829835.622373:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280936704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829835.622388:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829835.622397:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829835/real 1787829835] req@ffff8ab3f043a1c0 x1874671280936704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829835.622408:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280936704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829835.622417:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829835.622420:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829835.622423:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280936704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829835.622428:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280936832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829835.622441:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829835.622449:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829835/real 1787829835] req@ffff8ab3f043a940 x1874671280936832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829835.622459:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280936832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829835.622467:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829835.622470:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829835.622473:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280936832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829836.646044:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829837.670017:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829838.694091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829839.718054:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829840.742021:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829840.742032:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829840.742036:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829840.742047:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4390 00010000:00010000:3.0:1787829840.742050:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829840.742057:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829840.742065:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829840.742112:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280936960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829840.742138:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829840.742140:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829840.742142:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829840.742145:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4390 00000100:00080000:0.0:1787829840.742148:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829840.742152:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829840.742161:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829840.742173:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280937088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829840.742177:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280936960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829840.742181:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829840.742186:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829840.742188:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829840.742191:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4390 00000100:00080000:0.0:1787829840.742193:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829840.742195:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829840.742198:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829840.742206:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280937216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829840.742212:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829840.742215:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829840.742217:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829840.742221:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4390 00000100:00080000:0.0:1787829840.742244:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829840.742249:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829840.742254:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829840.742256:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829840.742268:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280937344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829840.742275:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829840.742278:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829840.742279:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829840.742282:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4390 00000100:00080000:0.0:1787829840.742285:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829840.742288:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829840.742291:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829840.742298:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280937472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829840.742316:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829840/real 1787829840] req@ffff8ab3f075e940 x1874671280936960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829840.742336:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280936960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829840.742346:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829840.742351:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829840.742355:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280936960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829840.742381:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280937088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829840.742397:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829840.742406:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829840/real 1787829840] req@ffff8ab3f075cb40 x1874671280937088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829840.742414:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280937088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829840.742419:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829840.742421:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829840.742423:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280937088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829840.742427:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280937216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829840.742436:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829840.742442:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829840/real 1787829840] req@ffff8ab3f075f0c0 x1874671280937216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829840.742448:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280937216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829840.742453:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829840.742454:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829840.742456:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280937216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829840.742459:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280937344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829840.742469:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829840.742475:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829840/real 1787829840] req@ffff8ab3f075de00 x1874671280937344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829840.742481:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280937344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829840.742487:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829840.742488:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829840.742491:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280937344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829840.742496:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280937472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829840.742512:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829840.742522:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829840/real 1787829840] req@ffff8ab3f075cf00 x1874671280937472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829840.742533:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280937472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829840.742542:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829840.742544:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829840.742546:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280937472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829841.766022:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829842.790089:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829843.813999:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829844.838060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829845.861987:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829845.861995:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829845.861998:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829845.862007:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4395 00000100:00080000:2.0:1787829845.862015:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829845.862021:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829845.862023:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829845.862052:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280937600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829845.862069:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829845.862072:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829845.862073:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829845.862075:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4395 00000100:00080000:2.0:1787829845.862077:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829845.862080:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829845.862087:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829845.862096:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280937728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829845.862104:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280937600:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829845.862105:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829845.862114:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829845.862116:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829845.862119:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4395 00000100:00080000:2.0:1787829845.862120:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829845.862123:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829845.862126:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829845.862132:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280937856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829845.862138:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829845.862141:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829845.862142:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829845.862144:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4395 00000100:00080000:2.0:1787829845.862154:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829845.862156:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829845.862159:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829845.862163:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829845.862170:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280937984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829845.862177:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829845.862179:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829845.862180:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829845.862182:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4395 00000100:00080000:2.0:1787829845.862184:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829845.862186:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829845.862190:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829845.862197:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280938112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829845.862207:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829845/real 1787829845] req@ffff8ab3f0439a40 x1874671280937600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829845.862226:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280937600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829845.862237:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829845.862239:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829845.862242:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280937600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829845.862259:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280937728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829845.862278:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829845.862291:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829845/real 1787829845] req@ffff8ab3f043bc00 x1874671280937728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829845.862303:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280937728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829845.862311:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829845.862314:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829845.862316:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280937728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829845.862327:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280937856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829845.862343:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829845.862351:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829845/real 1787829845] req@ffff8ab3f04383c0 x1874671280937856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829845.862358:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280937856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829845.862363:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829845.862365:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829845.862367:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280937856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829845.862371:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280937984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829845.862381:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829845.862386:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829845/real 1787829845] req@ffff8ab3f043ad00 x1874671280937984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829845.862393:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280937984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829845.862398:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829845.862400:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829845.862401:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280937984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829845.862404:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280938112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829845.862412:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829845.862417:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829845/real 1787829845] req@ffff8ab3f04392c0 x1874671280938112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829845.862424:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280938112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829845.862429:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829845.862431:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829845.862432:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280938112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829846.886060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829847.910073:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829848.934107:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829849.958098:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829850.982010:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829850.982018:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829850.982022:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829850.982031:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4400 00000100:00080000:0.0:1787829850.982040:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829850.982046:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829850.982056:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829850.982082:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280938240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829850.982103:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829850.982106:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829850.982108:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829850.982110:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4400 00000100:00080000:0.0:1787829850.982113:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829850.982116:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829850.982123:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829850.982136:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280938368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829850.982140:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280938240:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829850.982147:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829850.982150:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829850.982151:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829850.982154:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4400 00000100:00080000:0.0:1787829850.982156:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829850.982158:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829850.982161:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829850.982170:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280938496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829850.982176:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829850.982178:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829850.982180:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829850.982182:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4400 00000100:00080000:0.0:1787829850.982197:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829850.982200:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829850.982204:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829850.982208:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829850.982215:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280938624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829850.982223:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829850.982226:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829850.982228:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829850.982230:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4400 00000100:00080000:0.0:1787829850.982232:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829850.982234:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829850.982237:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829850.982245:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280938752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829850.982262:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829850/real 1787829850] req@ffff8ab3f075c000 x1874671280938240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829850.982283:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280938240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829850.982293:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829850.982296:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829850.982300:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280938240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829850.982325:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280938368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829850.982349:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829850.982362:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829850/real 1787829850] req@ffff8ab3f075cf00 x1874671280938368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829850.982372:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280938368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829850.982378:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829850.982381:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829850.982383:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280938368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829850.982387:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280938496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829850.982400:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829850.982406:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829850/real 1787829850] req@ffff8ab3f075de00 x1874671280938496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829850.982414:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280938496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829850.982424:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829850.982426:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829850.982428:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280938496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829850.982432:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280938624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829850.982441:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829850.982446:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829850/real 1787829850] req@ffff8ab3f075f0c0 x1874671280938624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829850.982454:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280938624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829850.982460:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829850.982462:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829850.982464:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280938624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829850.982468:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280938752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829850.982477:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829850.982482:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829850/real 1787829850] req@ffff8ab3f075cb40 x1874671280938752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829850.982488:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280938752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829850.982495:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829850.982497:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829850.982499:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280938752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829852.006092:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829853.030102:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829854.054098:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829855.078076:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829856.102009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829856.102024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829856.102029:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829856.102041:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4405 00000100:00080000:0.0:1787829856.102052:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829856.102068:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829856.102086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829856.102112:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280938880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829856.102145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829856.102149:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829856.102152:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829856.102158:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4405 00000100:00080000:0.0:1787829856.102163:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829856.102168:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829856.102181:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829856.102194:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280938880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829856.102200:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280939008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829856.102222:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829856.102228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829856.102230:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829856.102236:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4405 00000100:00080000:0.0:1787829856.102241:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829856.102247:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829856.102253:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829856.102269:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280939136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829856.102276:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829856.102294:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829856.102298:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829856.102300:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829856.102307:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4405 00000100:00080000:0.0:1787829856.102312:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829856.102319:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829856.102326:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829856.102335:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829856/real 1787829856] req@ffff8ab3f075e940 x1874671280938880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829856.102345:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280939264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829856.102364:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829856.102371:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829856.102371:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280938880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829856.102375:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829856.102385:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4405 00000100:00080000:0.0:1787829856.102388:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829856.102392:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829856.102403:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787829856.102418:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829856.102419:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280939392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829856.102437:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829856.102442:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280938880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829856.102468:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280939008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829856.102489:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829856.102499:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829856/real 1787829856] req@ffff8ab3f075f480 x1874671280939008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829856.102508:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280939008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829856.102515:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829856.102517:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829856.102520:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280939008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829856.102524:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280939136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829856.102535:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829856.102541:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829856/real 1787829856] req@ffff8ab3f075ed00 x1874671280939136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829856.102549:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280939136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829856.102555:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829856.102557:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829856.102559:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280939136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829856.102563:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280939264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829856.102573:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829856.102579:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829856/real 1787829856] req@ffff8ab3f075da40 x1874671280939264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829856.102586:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280939264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829856.102593:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829856.102595:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829856.102596:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280939264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829856.102600:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280939392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829856.102609:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829856.102615:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829856/real 1787829856] req@ffff8ab3f075f840 x1874671280939392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829856.102622:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280939392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829856.102629:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829856.102631:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829856.102632:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280939392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829857.126063:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829858.150169:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829859.174050:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829860.198088:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829861.221995:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829861.222009:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829861.222016:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829861.222030:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4410 00010000:00010000:3.0:1787829861.222035:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829861.222042:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829861.222058:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829861.222106:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280939520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829861.222131:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829861.222134:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829861.222140:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829861.222144:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4410 00000100:00080000:2.0:1787829861.222148:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829861.222153:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829861.222164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829861.222173:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280939520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829861.222180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280939648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829861.222192:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829861.222196:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829861.222198:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829861.222203:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4410 00000100:00080000:2.0:1787829861.222207:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829861.222211:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829861.222216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829861.222227:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280939776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829861.222235:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829861.222238:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829861.222240:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829861.222246:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4410 00000800:00000100:1.0:1787829861.222255:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829861.222256:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829861.222264:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829861.222270:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829861.222282:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280939904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829861.222289:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829861.222292:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829861.222295:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829861.222300:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4410 00000100:00080000:2.0:1787829861.222303:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829861.222308:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829861.222311:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829861/real 1787829861] req@ffff8ab3f0439680 x1874671280939520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829861.222313:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829861.222332:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280940032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829861.222341:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280939520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829861.222385:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829861.222391:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829861.222394:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280939520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829861.222417:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280939648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829861.222433:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829861.222442:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829861/real 1787829861] req@ffff8ab3f043b0c0 x1874671280939648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829861.222454:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280939648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829861.222463:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829861.222467:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829861.222470:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280939648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829861.222478:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280939776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829861.222507:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829861.222518:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829861/real 1787829861] req@ffff8ab3f043a580 x1874671280939776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829861.222526:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280939776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829861.222532:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829861.222535:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829861.222537:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280939776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829861.222542:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280939904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829861.222554:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829861.222560:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829861/real 1787829861] req@ffff8ab3f0438f00 x1874671280939904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829861.222568:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280939904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829861.222575:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829861.222577:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829861.222578:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280939904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829861.222582:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280940032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829861.222593:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829861.222599:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829861/real 1787829861] req@ffff8ab3f0438b40 x1874671280940032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829861.222607:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280940032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829861.222613:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829861.222615:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829861.222616:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280940032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829862.246109:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829863.270057:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829864.294094:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829865.318077:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829866.342076:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1787829866.342083:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829866.342084:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829866.342092:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829866.342103:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4415 00000100:00080000:0.0:1787829866.342112:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829866.342118:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829866.342154:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280940160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829866.342180:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829866.342182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829866.342184:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829866.342186:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4415 00000100:00080000:0.0:1787829866.342190:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829866.342194:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829866.342203:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829866.342218:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280940288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829866.342230:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829866.342233:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280940160:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829866.342236:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829866.342241:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829866.342247:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4415 00000100:00080000:0.0:1787829866.342251:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829866.342256:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829866.342261:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829866.342274:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280940416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829866.342281:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829866.342284:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829866.342285:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829866.342288:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4415 00000100:00080000:0.0:1787829866.342292:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829866.342294:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829866.342298:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829866.342304:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829866.342312:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280940544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829866.342323:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829866.342326:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829866.342327:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829866.342337:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4415 00000100:00080000:0.0:1787829866.342339:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829866.342342:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829866.342346:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829866.342352:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829866/real 1787829866] req@ffff8ab3f075e580 x1874671280940160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829866.342356:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280940672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829866.342388:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280940160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829866.342400:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829866.342405:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829866.342409:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280940160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829866.342433:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280940288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829866.342455:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829866.342466:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829866/real 1787829866] req@ffff8ab3f075f840 x1874671280940288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829866.342475:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280940288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829866.342486:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829866.342489:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829866.342491:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280940288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829866.342495:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280940416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829866.342507:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829866.342513:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829866/real 1787829866] req@ffff8ab3f075da40 x1874671280940416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829866.342520:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280940416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829866.342526:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829866.342527:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829866.342529:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280940416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829866.342533:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280940544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829866.342544:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829866.342551:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829866/real 1787829866] req@ffff8ab3f075ed00 x1874671280940544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829866.342558:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280940544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829866.342564:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829866.342565:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829866.342568:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280940544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829866.342572:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280940672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829866.342582:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829866.342587:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829866/real 1787829866] req@ffff8ab3f075f480 x1874671280940672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829866.342612:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280940672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829866.342619:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829866.342621:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829866.342624:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280940672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829867.366095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829868.390051:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829869.414091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829870.438080:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829871.462022:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829871.462034:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829871.462039:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829871.462050:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4421 00000100:00080000:2.0:1787829871.462061:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829871.462068:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829871.462110:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280940800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829871.462136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829871.462140:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829871.462141:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829871.462146:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4421 00000100:00080000:2.0:1787829871.462149:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1787829871.462151:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829871.462153:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829871.462171:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829871.462180:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280940800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829871.462192:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280940928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829871.462205:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829871.462209:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829871.462212:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829871.462217:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4421 00000100:00080000:2.0:1787829871.462237:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829871.462243:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829871.462248:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829871.462252:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829871.462270:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280941056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829871.462282:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829871.462286:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829871.462288:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829871.462299:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829871/real 1787829871] req@ffff8ab3f0438780 x1874671280940800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829871.462308:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4421 00000100:00080000:2.0:1787829871.462313:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829871.462316:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280940800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829871.462323:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787829871.462328:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829871.462332:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787829871.462333:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829871.462340:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280940800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829871.462353:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280941184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829871.462364:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280940928:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787829871.462368:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829871.462374:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829871.462376:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829871.462383:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829871.462383:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4421 00000100:00080000:2.0:1787829871.462392:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829871.462399:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829871/real 1787829871] req@ffff8ab3f0438b40 x1874671280940928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829871.462401:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829871.462413:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829871.462416:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280940928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829871.462424:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829871.462427:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829871.462432:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280940928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829871.462433:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280941312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829871.462438:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280941056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829871.462457:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829871.462465:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829871/real 1787829871] req@ffff8ab3f0438f00 x1874671280941056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829871.462474:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280941056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829871.462479:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829871.462481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829871.462484:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280941056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829871.462492:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280941184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829871.462503:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829871.462509:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829871/real 1787829871] req@ffff8ab3f043a580 x1874671280941184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829871.462516:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280941184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829871.462523:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829871.462525:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829871.462527:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280941184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829871.462530:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280941312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829871.462545:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829871.462566:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829871/real 1787829871] req@ffff8ab3f043b0c0 x1874671280941312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829871.462579:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280941312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829871.462591:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829871.462594:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829871.462597:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280941312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829872.486087:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829873.510137:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829874.534082:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829875.558093:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829876.582003:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829876.582018:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829876.582024:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829876.582037:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4426 00000100:00080000:2.0:1787829876.582050:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829876.582059:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829876.582117:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280941440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787829876.582126:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829876.582146:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829876.582151:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829876.582153:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829876.582157:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4426 00000100:00080000:2.0:1787829876.582161:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829876.582166:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829876.582179:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829876.582187:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280941440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829876.582196:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280941568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829876.582210:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829876.582215:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829876.582218:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829876.582224:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4426 00000100:00080000:2.0:1787829876.582252:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829876.582258:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829876.582263:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829876.582266:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829876.582285:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280941696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829876.582305:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829876.582309:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829876.582312:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829876.582315:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829876/real 1787829876] req@ffff8ab3f0439680 x1874671280941440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829876.582321:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4426 00000100:00080000:2.0:1787829876.582326:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829876.582334:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829876.582344:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829876.582357:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280941440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829876.582362:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280941824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829876.582384:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829876.582389:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829876.582390:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829876.582396:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829876.582397:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829876.582407:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280941440:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829876.582407:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4426 00000100:00080000:2.0:1787829876.582417:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829876.582425:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829876.582431:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829876.582439:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280941568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787829876.582447:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280941952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829876.582463:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829876.582476:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829876/real 1787829876] req@ffff8ab3f04392c0 x1874671280941568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829876.582486:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280941568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829876.582492:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829876.582494:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829876.582497:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280941568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829876.582500:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280941696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829876.582511:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829876.582529:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829876/real 1787829876] req@ffff8ab3f043ad00 x1874671280941696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829876.582544:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280941696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829876.582552:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829876.582556:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829876.582559:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280941696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829876.582565:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280941824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829876.582585:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829876.582598:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829876/real 1787829876] req@ffff8ab3f04383c0 x1874671280941824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829876.582608:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280941824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829876.582617:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829876.582620:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829876.582623:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280941824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829876.582639:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280941952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829876.582658:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829876.582669:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829876/real 1787829876] req@ffff8ab3f043bc00 x1874671280941952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829876.582681:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280941952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829876.582691:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829876.582695:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829876.582699:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280941952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829877.606074:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829878.630119:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829879.654073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829880.678018:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829881.702020:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829881.702031:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829881.702035:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829881.702048:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4431 00000100:00080000:0.0:1787829881.702060:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829881.702068:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829881.702068:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829881.702106:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280942080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829881.702129:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829881.702131:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829881.702133:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829881.702136:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4431 00000100:00080000:0.0:1787829881.702139:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829881.702144:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829881.702152:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829881.702162:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280942080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829881.702164:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280942208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829881.702177:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829881.702179:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829881.702180:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829881.702184:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4431 00000100:00080000:0.0:1787829881.702185:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829881.702188:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829881.702191:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829881.702201:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280942336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829881.702207:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829881.702210:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829881.702212:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829881.702214:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4431 00000100:00080000:0.0:1787829881.702217:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829881.702219:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829881.702223:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829881.702230:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280942464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829881.702236:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829881.702240:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829881.702242:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829881.702247:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4431 00000100:00080000:0.0:1787829881.702250:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829881.702255:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829881.702262:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829881.702273:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280942208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787829881.702280:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280942592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829881.702303:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280942336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829881.702321:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280942464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829881.702346:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280942592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829882.726069:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829883.750071:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829884.774075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787829884.774136:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787829884.774173:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829884.774186:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829884.774214:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829884.774221:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829884.774231:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787829884.774239:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787829884.774263:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829881/real 1787829884] req@ffff8ab3f075cb40 x1874671280942208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829884.774290:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280942208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829884.774303:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829884.774313:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829884.774319:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280942208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829884.774327:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829881/real 1787829884] req@ffff8ab3f075f0c0 x1874671280942336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829884.774342:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280942336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829884.774353:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829884.774357:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829884.774361:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280942336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829884.774368:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829881/real 1787829884] req@ffff8ab3f075de00 x1874671280942464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829884.774382:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280942464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829884.774393:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829884.774398:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829884.774402:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280942464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829884.774410:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829881/real 1787829884] req@ffff8ab3f075e940 x1874671280942080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829884.774423:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280942080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829884.774431:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829884.774435:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829884.774438:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280942080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829884.774475:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829881/real 1787829884] req@ffff8ab3f075cf00 x1874671280942592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829884.774490:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280942592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829884.774502:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829884.774505:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829884.774509:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280942592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829885.798098:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829886.822106:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829886.822134:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829886.822144:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829886.822150:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829886.822165:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4436 00000100:00080000:2.0:1787829886.822182:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829886.822192:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829886.822239:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280942720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829886.822272:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829886.822276:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829886.822278:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829886.822282:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4436 00000100:00080000:2.0:1787829886.822286:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829886.822292:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829886.822306:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829886.822324:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280942848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829886.822329:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280942720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829886.822344:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829886.822348:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829886.822351:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829886.822356:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4436 00000100:00080000:2.0:1787829886.822362:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829886.822366:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829886.822372:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829886.822387:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280942976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829886.822400:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829886.822404:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829886.822406:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829886.822412:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4436 00000100:00080000:2.0:1787829886.822429:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829886.822436:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829886.822442:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829886.822461:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829886.822471:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280943104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829886.822484:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829886.822489:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829886.822491:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829886.822501:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4436 00000100:00080000:2.0:1787829886.822506:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829886.822512:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829886.822518:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829886.822536:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829886/real 1787829886] req@ffff8ab3f0439a40 x1874671280942720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829886.822536:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280943232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829886.822582:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280942720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829886.822598:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829886.822603:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829886.822608:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280942720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829886.822641:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280942848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829886.822667:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829886.822683:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829886/real 1787829886] req@ffff8ab3f043a940 x1874671280942848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829886.822697:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280942848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829886.822706:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829886.822710:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829886.822713:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280942848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829886.822721:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280942976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829886.822741:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829886.822753:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829886/real 1787829886] req@ffff8ab3f043a1c0 x1874671280942976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829886.822765:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280942976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829886.822773:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829886.822776:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829886.822780:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280942976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829886.822786:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280943104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829886.822806:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829886.822818:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829886/real 1787829886] req@ffff8ab3f0439e00 x1874671280943104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829886.822829:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280943104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829886.822841:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829886.822844:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829886.822848:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280943104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829886.822854:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280943232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829886.822875:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829886.822909:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829886/real 1787829886] req@ffff8ab3f043b480 x1874671280943232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829886.822922:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280943232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829886.822933:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829886.822937:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829886.822940:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280943232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829887.846147:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829888.870119:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829889.894071:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829890.918110:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829891.941987:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829891.941998:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829891.942003:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829891.942014:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4441 00000100:00080000:2.0:1787829891.942025:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829891.942033:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829891.942067:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829891.942084:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280943360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829891.942107:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829891.942110:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829891.942112:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829891.942115:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4441 00000100:00080000:2.0:1787829891.942118:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829891.942121:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829891.942131:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829891.942142:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280943488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829891.942150:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280943360:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829891.942153:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829891.942159:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829891.942161:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829891.942165:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4441 00000100:00080000:2.0:1787829891.942167:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829891.942169:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829891.942174:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829891.942183:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280943616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829891.942189:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829891.942192:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829891.942193:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829891.942196:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4441 00000100:00080000:2.0:1787829891.942215:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829891.942220:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829891.942225:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:2.0:1787829891.942227:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829891.942245:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280943744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829891.942257:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829891.942262:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829891.942264:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829891.942268:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4441 00000100:00080000:2.0:1787829891.942273:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829891.942277:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829891.942280:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829891/real 1787829891] req@ffff8ab3f043b840 x1874671280943360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829891.942283:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829891.942299:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280943872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829891.942304:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280943360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829891.942320:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829891.942324:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829891.942328:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280943360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829891.942350:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280943488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829891.942365:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829891.942374:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829891/real 1787829891] req@ffff8ab3f043b480 x1874671280943488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829891.942382:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280943488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829891.942388:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829891.942391:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829891.942393:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280943488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829891.942397:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280943616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829891.942406:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829891.942425:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829891/real 1787829891] req@ffff8ab3f0439e00 x1874671280943616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829891.942435:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280943616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829891.942445:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829891.942447:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829891.942449:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280943616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829891.942453:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280943744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829891.942466:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829891.942472:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829891/real 1787829891] req@ffff8ab3f043a1c0 x1874671280943744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829891.942479:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280943744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829891.942486:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829891.942488:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829891.942490:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280943744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829891.942494:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280943872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829891.942503:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829891.942509:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829891/real 1787829891] req@ffff8ab3f043a940 x1874671280943872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829891.942516:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280943872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829891.942522:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829891.942524:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829891.942527:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280943872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829892.966115:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829893.990062:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829895.014080:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829896.038079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829897.062014:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829897.062022:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829897.062026:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829897.062034:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4446 00000100:00080000:2.0:1787829897.062044:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829897.062050:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829897.062084:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280944000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787829897.062103:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829897.062104:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829897.062109:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829897.062110:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829897.062114:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4446 00000100:00080000:2.0:1787829897.062117:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829897.062120:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829897.062128:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829897.062139:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280944000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829897.062139:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280944128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829897.062153:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829897.062156:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829897.062157:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829897.062160:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4446 00000100:00080000:2.0:1787829897.062162:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829897.062165:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829897.062168:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829897.062178:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280944256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829897.062186:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829897.062188:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829897.062189:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829897.062192:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4446 00000100:00080000:2.0:1787829897.062194:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829897.062199:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829897.062200:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829897.062206:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829897.062215:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280944384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829897.062221:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829897.062223:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829897.062225:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829897.062227:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4446 00000100:00080000:2.0:1787829897.062230:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829897.062233:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829897.062237:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829897.062244:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829897/real 1787829897] req@ffff8ab3f0439a40 x1874671280944000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829897.062246:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280944512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829897.062276:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280944000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829897.062287:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829897.062290:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829897.062293:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280944000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829897.062311:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280944128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829897.062325:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829897.062333:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829897/real 1787829897] req@ffff8ab3f043bc00 x1874671280944128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829897.062341:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280944128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829897.062347:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829897.062349:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829897.062351:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280944128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829897.062355:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280944256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829897.062364:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829897.062370:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829897/real 1787829897] req@ffff8ab3f04383c0 x1874671280944256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829897.062400:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280944256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829897.062406:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829897.062408:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829897.062411:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280944256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829897.062415:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280944384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829897.062433:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829897.062444:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829897/real 1787829897] req@ffff8ab3f043ad00 x1874671280944384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829897.062457:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280944384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829897.062467:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829897.062471:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829897.062475:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280944384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829897.062480:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280944512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829897.062495:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829897.062502:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829897/real 1787829897] req@ffff8ab3f04392c0 x1874671280944512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829897.062510:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280944512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829897.062517:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829897.062519:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829897.062521:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280944512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829898.086045:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829899.110072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829900.134046:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829901.158070:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829902.182002:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829902.182019:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829902.182023:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829902.182032:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4451 00000100:00080000:0.0:1787829902.182043:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829902.182051:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829902.182058:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829902.182088:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280944640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829902.182113:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829902.182116:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829902.182117:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829902.182121:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4451 00000100:00080000:0.0:1787829902.182125:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829902.182130:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829902.182139:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829902.182153:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280944640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829902.182157:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280944768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829902.182184:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829902.182187:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829902.182189:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829902.182194:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4451 00000100:00080000:0.0:1787829902.182198:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829902.182203:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829902.182209:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829902.182223:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280944896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829902.182232:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829902.182241:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829902.182246:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829902.182249:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829902.182257:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4451 00000100:00080000:0.0:1787829902.182261:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829902.182267:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829902.182274:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829902.182286:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829902/real 1787829902] req@ffff8ab3f075c000 x1874671280944640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829902.182293:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280945024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829902.182312:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829902.182313:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280944640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829902.182330:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829902.182338:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829902.182339:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829902.182357:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787829902.182358:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4451 00000100:00100000:1.0:1787829902.182367:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280944640:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829902.182368:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829902.182378:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829902.182383:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829902.182396:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280945152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829902.182406:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280944768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829902.182429:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829902.182439:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829902/real 1787829902] req@ffff8ab3f075cf00 x1874671280944768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829902.182446:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280944768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829902.182452:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829902.182454:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829902.182456:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280944768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829902.182460:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280944896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829902.182470:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829902.182475:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829902/real 1787829902] req@ffff8ab3f075e940 x1874671280944896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829902.182482:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280944896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829902.182486:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829902.182488:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829902.182489:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280944896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829902.182493:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280945024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829902.182508:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829902.182518:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829902/real 1787829902] req@ffff8ab3f075de00 x1874671280945024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829902.182529:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280945024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829902.182540:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829902.182543:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829902.182546:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280945024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829902.182558:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280945152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829902.182572:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829902.182579:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829902/real 1787829902] req@ffff8ab3f075f0c0 x1874671280945152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829902.182597:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280945152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829902.182604:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829902.182606:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829902.182608:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280945152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829903.206078:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829904.230122:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829905.254088:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829906.278087:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829907.302048:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829907.302059:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829907.302064:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829907.302078:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4456 00000100:00080000:0.0:1787829907.302090:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787829907.302091:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829907.302099:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829907.302137:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280945280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829907.302164:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829907.302168:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829907.302170:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829907.302175:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4456 00000100:00080000:0.0:1787829907.302180:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829907.302185:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829907.302194:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829907.302207:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280945280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829907.302211:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280945408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829907.302231:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829907.302235:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829907.302237:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829907.302243:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4456 00000100:00080000:0.0:1787829907.302246:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829907.302251:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829907.302257:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829907.302271:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280945536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829907.302281:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829907.302285:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829907.302287:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829907.302296:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829907.302301:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4456 00000100:00080000:0.0:1787829907.302319:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829907.302325:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829907.302331:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829907.302344:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280945664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829907.302356:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829907.302360:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829907.302362:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829907.302365:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829907/real 1787829907] req@ffff8ab3f075cb40 x1874671280945280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829907.302380:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4456 00000100:00080000:0.0:1787829907.302389:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829907.302394:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280945280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829907.302396:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829907.302409:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787829907.302415:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829907.302421:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787829907.302427:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280945792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829907.302429:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280945280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829907.302472:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280945408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829907.302496:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829907.302509:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829907/real 1787829907] req@ffff8ab3f075f480 x1874671280945408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829907.302520:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280945408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829907.302535:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829907.302538:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829907.302541:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280945408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829907.302547:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280945536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829907.302562:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829907.302571:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829907/real 1787829907] req@ffff8ab3f075ed00 x1874671280945536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829907.302581:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280945536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829907.302588:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829907.302591:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829907.302594:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280945536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829907.302600:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280945664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829907.302618:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829907.302629:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829907/real 1787829907] req@ffff8ab3f075da40 x1874671280945664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829907.302640:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280945664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829907.302649:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829907.302652:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829907.302655:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280945664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829907.302661:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280945792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829907.302680:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829907.302690:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829907/real 1787829907] req@ffff8ab3f075f840 x1874671280945792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829907.302702:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280945792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829907.302719:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829907.302722:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829907.302726:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280945792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829908.326198:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829909.350041:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829910.374130:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829911.398096:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829912.422059:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829912.422073:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1787829912.422078:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829912.422079:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829912.422093:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4462 00000100:00080000:2.0:1787829912.422105:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829912.422124:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829912.422174:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280945920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829912.422204:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829912.422208:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829912.422210:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829912.422214:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4462 00000100:00080000:2.0:1787829912.422218:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829912.422224:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829912.422237:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829912.422253:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280945920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829912.422264:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280946048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829912.422290:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829912.422294:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829912.422296:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829912.422302:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4462 00000100:00080000:2.0:1787829912.422306:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829912.422311:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829912.422317:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829912.422333:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280946176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829912.422345:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829912.422358:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829912.422362:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829912.422365:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829912.422371:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4462 00000100:00080000:2.0:1787829912.422376:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829912.422382:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829912.422387:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829912.422399:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829912/real 1787829912] req@ffff8ab3f0439680 x1874671280945920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829912.422405:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280946304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829912.422420:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829912.422430:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829912.422432:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829912.422439:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4462 00000100:00100000:1.0:1787829912.422445:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280945920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829912.422447:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829912.422460:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787829912.422470:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829912.422471:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787829912.422481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829912.422489:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280945920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829912.422493:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280946432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829912.422524:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280946048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829912.422559:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829912.422574:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829912/real 1787829912] req@ffff8ab3f043b0c0 x1874671280946048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829912.422586:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280946048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829912.422593:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829912.422597:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829912.422600:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280946048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829912.422606:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280946176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829912.422624:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829912.422633:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829912/real 1787829912] req@ffff8ab3f043a580 x1874671280946176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829912.422644:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280946176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829912.422652:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829912.422656:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829912.422658:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280946176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829912.422665:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280946304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829912.422683:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829912.422692:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829912/real 1787829912] req@ffff8ab3f0438f00 x1874671280946304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829912.422703:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280946304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829912.422712:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829912.422715:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829912.422718:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280946304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829912.422723:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280946432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829912.422740:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829912.422750:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829912/real 1787829912] req@ffff8ab3f0438b40 x1874671280946432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829912.422760:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280946432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829912.422768:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829912.422771:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829912.422774:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280946432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829913.446075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829914.470069:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829915.494047:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829916.518069:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829917.541960:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829917.541970:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829917.541974:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829917.541984:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4467 00000100:00080000:2.0:1787829917.541994:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829917.542001:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787829917.542012:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829917.542035:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280946560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829917.542058:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829917.542062:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829917.542064:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829917.542068:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4467 00000100:00080000:2.0:1787829917.542071:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829917.542076:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829917.542084:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829917.542085:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280946560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829917.542101:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280946688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829917.542111:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829917.542115:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829917.542117:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829917.542121:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4467 00000100:00080000:2.0:1787829917.542131:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829917.542136:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829917.542137:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829917.542145:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829917.542157:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280946816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829917.542172:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829917.542175:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829917/real 1787829917] req@ffff8ab3f0438780 x1874671280946560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829917.542177:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829917.542181:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829917.542190:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4467 00000100:00080000:2.0:1787829917.542194:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829917.542198:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280946560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829917.542201:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829917.542210:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787829917.542213:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829917.542218:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829917.542225:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280946560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829917.542225:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280946944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829917.542251:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829917.542255:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829917.542257:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829917.542262:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4467 00000100:00100000:1.0:1787829917.542265:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280946688:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787829917.542267:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829917.542273:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829917.542278:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787829917.542285:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787829917.542291:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280947072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829917.542297:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829917/real 1787829917] req@ffff8ab3f0438b40 x1874671280946688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829917.542320:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280946688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829917.542326:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829917.542329:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829917.542331:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280946688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829917.542336:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280946816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829917.542346:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829917.542351:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829917/real 1787829917] req@ffff8ab3f0438f00 x1874671280946816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829917.542357:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280946816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829917.542362:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829917.542363:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829917.542365:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280946816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829917.542368:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280946944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829917.542377:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829917.542382:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829917/real 1787829917] req@ffff8ab3f043a580 x1874671280946944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829917.542389:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280946944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829917.542395:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829917.542396:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829917.542398:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280946944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829917.542406:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280947072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829917.542414:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829917.542419:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829917/real 1787829917] req@ffff8ab3f043b0c0 x1874671280947072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829917.542426:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280947072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829917.542438:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829917.542439:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829917.542441:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280947072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829918.566075:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829919.589984:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829920.614077:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829921.638086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829922.662002:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829922.662013:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829922.662018:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829922.662027:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4472 00000100:00080000:0.0:1787829922.662038:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829922.662046:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829922.662055:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829922.662094:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280947200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829922.662120:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829922.662123:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829922.662124:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829922.662126:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4472 00000100:00080000:0.0:1787829922.662129:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829922.662131:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829922.662141:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829922.662155:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280947328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829922.662163:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829922.662166:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829922.662168:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829922.662173:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4472 00000100:00100000:1.0:1787829922.662174:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280947200:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829922.662176:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829922.662183:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829922.662191:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829922.662211:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280947456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829922.662223:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829922.662226:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829922.662228:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829922.662233:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4472 00000100:00080000:0.0:1787829922.662237:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829922.662240:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829922.662244:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829922.662256:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280947584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829922.662265:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829922.662273:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829922.662274:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829922.662279:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829922.662285:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4472 00000100:00080000:0.0:1787829922.662291:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829922.662295:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829922.662301:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829922.662313:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280947712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829922.662341:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829922/real 1787829922] req@ffff8ab3f075e580 x1874671280947200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829922.662360:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280947200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829922.662372:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829922.662377:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829922.662381:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280947200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829922.662406:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280947328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829922.662431:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829922.662444:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829922/real 1787829922] req@ffff8ab3f075f840 x1874671280947328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829922.662457:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280947328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829922.662474:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829922.662478:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829922.662481:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280947328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829922.662489:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280947456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829922.662509:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829922.662520:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829922/real 1787829922] req@ffff8ab3f075da40 x1874671280947456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829922.662533:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280947456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829922.662541:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829922.662546:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829922.662550:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280947456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829922.662557:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280947584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829922.662578:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829922.662590:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829922/real 1787829922] req@ffff8ab3f075ed00 x1874671280947584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829922.662602:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280947584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829922.662612:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829922.662615:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829922.662619:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280947584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829922.662625:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280947712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829922.662644:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829922.662655:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829922/real 1787829922] req@ffff8ab3f075f480 x1874671280947712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829922.662667:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280947712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829922.662677:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829922.662680:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829922.662683:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280947712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829923.686088:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829924.710103:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829925.734064:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829926.758088:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829927.781996:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829927.782010:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829927.782015:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829927.782034:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4477 00000100:00080000:0.0:1787829927.782046:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829927.782055:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829927.782058:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829927.782103:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280947840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829927.782135:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829927.782138:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829927.782141:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829927.782145:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4477 00000100:00080000:0.0:1787829927.782157:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829927.782174:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280947840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829927.782176:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829927.782189:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829927.782202:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280947968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829927.782213:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829927.782216:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829927.782217:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829927.782220:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4477 00000100:00080000:0.0:1787829927.782232:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829927.782235:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829927.782237:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829927.782241:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829927.782247:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280948096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829927.782258:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829927.782268:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829927.782270:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829927.782275:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4477 00000100:00080000:0.0:1787829927.782280:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829927.782285:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829927.782289:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829927/real 1787829927] req@ffff8ab3f075cb40 x1874671280947840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829927.782291:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829927.782310:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280947840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829927.782311:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280948224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829927.782331:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829927.782332:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829927.782337:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829927.782339:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:0.0:1787829927.782340:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829927.782345:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4477 00000100:00100000:1.0:1787829927.782345:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280947840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829927.782351:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829927.782355:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829927.782358:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829927.782365:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280948352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829927.782371:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280947968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829927.782398:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829927.782408:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829927/real 1787829927] req@ffff8ab3f075f0c0 x1874671280947968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829927.782415:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280947968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829927.782420:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829927.782422:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829927.782424:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280947968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829927.782427:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280948096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829927.782436:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829927.782441:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829927/real 1787829927] req@ffff8ab3f075de00 x1874671280948096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829927.782447:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280948096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829927.782456:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829927.782458:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829927.782460:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280948096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829927.782463:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280948224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829927.782472:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829927.782477:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829927/real 1787829927] req@ffff8ab3f075e940 x1874671280948224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829927.782482:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280948224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829927.782489:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829927.782490:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829927.782492:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280948224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829927.782501:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280948352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829927.782510:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829927.782515:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829927/real 1787829927] req@ffff8ab3f075cf00 x1874671280948352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829927.782521:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280948352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829927.782527:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829927.782528:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829927.782530:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280948352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829928.806047:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829929.830077:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829930.854097:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829931.878066:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829932.901991:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829932.901999:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829932.902003:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829932.902011:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4482 00000100:00080000:0.0:1787829932.902019:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829932.902025:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829932.902050:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829932.902055:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280948480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829932.902074:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829932.902078:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829932.902080:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829932.902083:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4482 00000100:00080000:0.0:1787829932.902085:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829932.902088:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829932.902095:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829932.902101:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280948480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829932.902110:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280948608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829932.902122:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829932.902126:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829932.902142:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829932.902147:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4482 00000100:00080000:0.0:1787829932.902158:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829932.902163:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829932.902166:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829932.902172:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829932.902186:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280948736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829932.902203:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829932.902204:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829932/real 1787829932] req@ffff8ab3f075c000 x1874671280948480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829932.902210:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829932.902214:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829932.902220:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4482 00000100:00100000:1.0:1787829932.902222:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280948480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829932.902227:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829932.902236:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787829932.902236:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829932.902244:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000008:00100000:0.0:1787829932.902245:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829932.902250:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280948480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829932.902262:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280948864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829932.902274:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829932.902276:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280948608:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787829932.902279:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829932.902283:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829932.902288:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4482 00000100:00080000:0.0:1787829932.902295:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829932.902298:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829932.902303:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829932.902309:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829932.902312:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829932/real 1787829932] req@ffff8ab3f075cf00 x1874671280948608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829932.902323:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280948992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829932.902329:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280948608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829932.902342:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829932.902346:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829932.902350:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280948608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829932.902355:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280948736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829932.902369:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829932.902377:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829932/real 1787829932] req@ffff8ab3f075e940 x1874671280948736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829932.902384:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280948736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829932.902389:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829932.902391:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829932.902394:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280948736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829932.902406:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280948864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829932.902424:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829932.902435:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829932/real 1787829932] req@ffff8ab3f075de00 x1874671280948864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829932.902446:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280948864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829932.902452:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829932.902454:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829932.902456:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280948864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829932.902460:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280948992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829932.902473:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829932.902478:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829932/real 1787829932] req@ffff8ab3f075f0c0 x1874671280948992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829932.902488:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280948992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829932.902497:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829932.902500:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829932.902503:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280948992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829933.926133:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829934.950101:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829935.974677:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829936.998025:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829938.022082:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829938.022106:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829938.022112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829938.022117:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829938.022126:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4487 00000100:00080000:0.0:1787829938.022135:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829938.022140:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829938.022173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280949120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829938.022198:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829938.022201:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829938.022203:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829938.022206:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4487 00000100:00080000:0.0:1787829938.022209:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829938.022212:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829938.022221:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829938.022232:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280949248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829938.022251:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280949120:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829938.022301:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829938.022308:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1787829938.022308:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829938.022312:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829938.022319:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4487 00000100:00080000:0.0:1787829938.022324:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829938.022329:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829938.022335:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829938.022350:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280949376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829938.022359:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829938/real 1787829938] req@ffff8ab3f075cb40 x1874671280949120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829938.022362:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829938.022369:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829938.022374:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829938.022382:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4487 00000100:00080000:0.0:1787829938.022386:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829938.022388:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280949120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829938.022393:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829938.022403:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787829938.022407:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829938.022414:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787829938.022419:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280949504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829938.022420:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280949120:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829938.022440:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829938.022443:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829938.022445:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829938.022449:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280949248:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787829938.022452:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4487 00000100:00080000:0.0:1787829938.022458:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829938.022463:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829938.022468:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787829938.022477:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829938.022481:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280949632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829938.022493:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829938/real 1787829938] req@ffff8ab3f075f480 x1874671280949248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829938.022507:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280949248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829938.022516:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829938.022519:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829938.022523:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280949248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829938.022529:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280949376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829938.022554:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829938.022563:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829938/real 1787829938] req@ffff8ab3f075ed00 x1874671280949376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829938.022574:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280949376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829938.022582:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829938.022585:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829938.022588:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280949376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829938.022594:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280949504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829938.022608:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829938.022617:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829938/real 1787829938] req@ffff8ab3f075da40 x1874671280949504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829938.022627:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280949504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829938.022636:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829938.022639:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829938.022642:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280949504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829938.022655:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280949632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829938.022669:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829938.022678:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829938/real 1787829938] req@ffff8ab3f075f840 x1874671280949632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829938.022697:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280949632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829938.022708:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829938.022711:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829938.022714:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280949632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829939.046056:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829940.070095:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829941.094010:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829942.118100:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829943.142020:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829943.142036:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829943.142041:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829943.142069:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4492 00000100:00080000:0.0:1787829943.142082:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829943.142092:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829943.142112:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829943.142143:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280949760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829943.142172:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829943.142176:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829943.142178:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829943.142183:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4492 00000100:00080000:0.0:1787829943.142187:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829943.142192:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829943.142206:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829943.142210:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280949760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829943.142224:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280949888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829943.142235:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829943.142238:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829943.142240:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829943.142244:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4492 00000100:00080000:0.0:1787829943.142248:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829943.142252:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829943.142257:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829943.142270:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280950016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829943.142280:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829943.142283:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829943.142285:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829943.142289:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4492 00000100:00080000:0.0:1787829943.142294:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829943.142295:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787829943.142303:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829943.142308:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829943.142318:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280950144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829943.142334:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829943.142341:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829943.142344:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829943.142349:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4492 00000100:00080000:0.0:1787829943.142354:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829943.142359:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829943.142365:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829943.142366:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829943/real 1787829943] req@ffff8ab3f075e580 x1874671280949760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829943.142380:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280950272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829943.142389:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280949760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829943.142398:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829943.142401:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829943.142406:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280949760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829943.142430:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280949888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829943.142446:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829943.142456:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829943/real 1787829943] req@ffff8ab3f075f840 x1874671280949888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829943.142471:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280949888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829943.142480:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829943.142482:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829943.142484:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280949888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829943.142487:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280950016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829943.142498:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829943.142503:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829943/real 1787829943] req@ffff8ab3f075da40 x1874671280950016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829943.142509:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280950016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829943.142514:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829943.142516:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829943.142518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280950016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829943.142521:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280950144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829943.142530:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829943.142535:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829943/real 1787829943] req@ffff8ab3f075ed00 x1874671280950144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829943.142544:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280950144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829943.142554:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829943.142557:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829943.142560:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280950144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829943.142565:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280950272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829943.142584:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829943.142601:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829943/real 1787829943] req@ffff8ab3f075f480 x1874671280950272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829943.142608:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280950272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829943.142614:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829943.142616:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829943.142618:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280950272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829944.166072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829945.190187:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829946.214078:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829947.238100:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829948.262004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829948.262014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829948.262018:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829948.262037:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4497 00000100:00080000:0.0:1787829948.262048:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787829948.262050:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829948.262055:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829948.262095:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280950400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829948.262115:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829948.262117:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829948.262119:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829948.262121:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4497 00000100:00080000:0.0:1787829948.262124:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829948.262126:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829948.262135:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829948.262145:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280950528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829948.262152:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280950400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787829948.262155:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829948.262160:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829948.262161:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829948.262165:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4497 00000100:00080000:0.0:1787829948.262167:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829948.262169:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829948.262172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829948.262181:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280950656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829948.262187:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829948.262190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829948.262191:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829948.262193:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4497 00000100:00080000:0.0:1787829948.262214:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829948.262217:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829948.262219:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829948.262227:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280950784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829948.262233:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829948.262235:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829948.262236:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829948.262239:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4497 00000100:00080000:0.0:1787829948.262242:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829948.262245:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829948.262249:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829948.262258:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280950912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829948.262275:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280950528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829948.262304:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280950656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829948.262321:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280950784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829948.262339:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280950912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829949.286067:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829950.310049:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829951.334190:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829951.334302:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787829951.334359:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829951.334391:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829951.334446:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829951.334459:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829951.334472:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787829951.334485:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787829951.334528:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829948/real 1787829951] req@ffff8ab3f075f0c0 x1874671280950528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829951.334570:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280950528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829951.334591:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829951.334596:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829951.334601:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280950528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829951.334610:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829948/real 1787829951] req@ffff8ab3f075de00 x1874671280950656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829951.334623:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280950656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829951.334632:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829951.334636:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829951.334640:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280950656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829951.334648:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829948/real 1787829951] req@ffff8ab3f075e940 x1874671280950784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829951.334662:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280950784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829951.334673:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829951.334677:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829951.334681:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280950784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829951.334695:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829948/real 1787829951] req@ffff8ab3f075cf00 x1874671280950912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829951.334707:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280950912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829951.334719:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829951.334722:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829951.334725:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280950912:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829951.334732:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829948/real 1787829951] req@ffff8ab3f075cb40 x1874671280950400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829951.334743:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280950400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829951.334750:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829951.334754:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829951.334757:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280950400:192.168.203.124@tcp:250:kworker.0 00010000:00010000:2.0:1787829952.358090:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829953.382088:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829953.382103:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1787829953.382106:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829953.382108:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829953.382122:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4503 00000100:00080000:3.0:1787829953.382142:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829953.382150:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829953.382192:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280951040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829953.382223:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829953.382227:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829953.382229:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829953.382233:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4503 00000100:00080000:3.0:1787829953.382236:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829953.382240:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829953.382252:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829953.382270:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280951168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829953.382279:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829953.382282:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829953.382285:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829953.382289:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4503 00000100:00080000:3.0:1787829953.382297:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829953.382301:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829953.382306:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829953.382317:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280951296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829953.382326:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829953.382329:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829953.382331:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829953.382335:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4503 00000100:00080000:3.0:1787829953.382339:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829953.382343:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829953.382347:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829953.382359:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280951424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829953.382367:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829953.382370:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829953.382371:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829953.382375:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4503 00000100:00080000:3.0:1787829953.382378:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829953.382381:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829953.382385:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829953.382397:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280951552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829953.382478:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280951040:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787829953.382556:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829953.382612:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829953/real 1787829953] req@ffff8ab3d0a43480 x1874671280951040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829953.382635:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280951040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829953.382645:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829953.382649:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829953.382653:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280951040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829953.382660:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280951168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829953.382683:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829953.382696:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829953/real 1787829953] req@ffff8ab3d0a43840 x1874671280951168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829953.382706:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280951168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829953.382712:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829953.382715:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829953.382718:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280951168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829953.382724:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280951296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829953.382739:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829953.382749:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829953/real 1787829953] req@ffff8ab3d0a412c0 x1874671280951296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829953.382758:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280951296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829953.382765:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829953.382768:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829953.382771:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280951296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829953.382776:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280951424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829953.382794:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829953.382804:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829953/real 1787829953] req@ffff8ab3d0a430c0 x1874671280951424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829953.382820:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280951424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829953.382829:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829953.382832:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829953.382835:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280951424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829953.382840:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280951552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829953.382857:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829953.382868:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829953/real 1787829953] req@ffff8ab3d0a41e00 x1874671280951552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829953.382877:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280951552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829953.382904:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829953.382908:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829953.382911:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280951552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829954.406070:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829955.430104:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829956.454163:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829957.478116:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829958.502045:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1787829958.502055:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829958.502056:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829958.502062:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829958.502081:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4508 00000100:00080000:2.0:1787829958.502095:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829958.502102:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829958.502141:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280951680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829958.502166:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829958.502170:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829958.502172:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829958.502175:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4508 00000100:00080000:2.0:1787829958.502180:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829958.502185:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829958.502194:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829958.502210:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280951808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829958.502217:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280951680:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829958.502230:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829958.502239:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829958.502242:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829958.502255:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4508 00000100:00080000:2.0:1787829958.502259:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829958.502267:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829958.502273:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829958.502288:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280951936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829958.502297:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829958.502301:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829958.502303:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829958.502307:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4508 00000100:00080000:2.0:1787829958.502310:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829958.502318:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829958.502319:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829958.502330:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829958.502345:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280952064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829958.502356:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829958.502359:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829958.502361:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829958.502367:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4508 00000100:00080000:2.0:1787829958.502371:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829958.502376:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829958.502382:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829958.502384:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829958/real 1787829958] req@ffff8ab3f0439680 x1874671280951680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829958.502399:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280952192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829958.502411:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280951680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829958.502430:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829958.502435:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829958.502440:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280951680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829958.502469:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280951808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829958.502490:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829958.502502:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829958/real 1787829958] req@ffff8ab3f04392c0 x1874671280951808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829958.502512:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280951808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829958.502519:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829958.502522:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829958.502525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280951808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829958.502530:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280951936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829958.502545:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829958.502553:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829958/real 1787829958] req@ffff8ab3f043ad00 x1874671280951936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829958.502563:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280951936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829958.502570:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829958.502573:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829958.502576:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280951936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829958.502582:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280952064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829958.502597:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829958.502606:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829958/real 1787829958] req@ffff8ab3f04383c0 x1874671280952064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829958.502616:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280952064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829958.502625:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829958.502627:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829958.502630:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280952064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829958.502635:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280952192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829958.502651:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829958.502661:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829958/real 1787829958] req@ffff8ab3f043bc00 x1874671280952192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829958.502670:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280952192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829958.502679:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829958.502682:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829958.502685:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280952192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829959.526123:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829960.550079:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829961.574140:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829962.598198:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829963.622005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829963.622019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829963.622024:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829963.622036:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4513 00000100:00080000:2.0:1787829963.622053:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829963.622062:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787829963.622073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829963.622123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280952320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829963.622149:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829963.622152:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829963.622153:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829963.622161:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4513 00000100:00080000:2.0:1787829963.622165:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829963.622169:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829963.622180:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829963.622199:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280952448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829963.622200:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280952320:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829963.622222:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829963.622226:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829963.622228:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829963.622232:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4513 00000100:00080000:2.0:1787829963.622236:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829963.622241:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829963.622246:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829963.622259:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280952576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829963.622278:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829963.622282:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829963.622284:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829963.622289:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4513 00000800:00000100:1.0:1787829963.622294:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829963.622295:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829963.622303:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829963.622309:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829963.622321:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280952704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829963.622340:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829963.622343:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829963.622344:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829963.622348:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4513 00000100:00080000:2.0:1787829963.622352:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829963.622356:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829963.622362:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829963/real 1787829963] req@ffff8ab3f0439a40 x1874671280952320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829963.622362:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829963.622378:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280952832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829963.622385:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280952320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829963.622406:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829963.622411:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829963.622415:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280952320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829963.622440:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280952448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829963.622467:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829963.622481:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829963/real 1787829963] req@ffff8ab3f043a940 x1874671280952448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829963.622505:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280952448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829963.622513:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829963.622516:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829963.622520:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280952448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829963.622525:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280952576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829963.622544:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829963.622555:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829963/real 1787829963] req@ffff8ab3f043a1c0 x1874671280952576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829963.622565:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280952576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829963.622580:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829963.622583:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829963.622586:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280952576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829963.622592:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280952704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829963.622612:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829963.622624:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829963/real 1787829963] req@ffff8ab3f0439e00 x1874671280952704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829963.622634:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280952704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829963.622643:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829963.622647:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829963.622649:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280952704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829963.622655:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280952832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829963.622671:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829963.622682:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829963/real 1787829963] req@ffff8ab3f043b480 x1874671280952832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829963.622699:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280952832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829963.622712:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829963.622715:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829963.622719:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280952832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829964.646104:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829965.670094:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829966.694877:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829967.718959:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829968.742014:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829968.742024:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829968.742028:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829968.742038:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4518 00000100:00080000:2.0:1787829968.742048:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829968.742055:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829968.742093:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280952960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829968.742119:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829968.742122:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829968.742124:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829968.742128:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4518 00000100:00080000:2.0:1787829968.742132:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829968.742136:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829968.742144:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829968.742157:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280953088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829968.742167:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829968.742172:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829968.742174:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829968.742177:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4518 00000100:00080000:2.0:1787829968.742181:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829968.742185:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829968.742190:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829968.742200:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280953216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829968.742208:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280952960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787829968.742213:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829968.742217:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829968.742219:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829968.742223:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4518 00000100:00080000:2.0:1787829968.742227:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829968.742231:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829968.742236:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787829968.742247:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280953344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829968.742256:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829968.742259:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829968.742261:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829968.742266:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4518 00000800:00000100:1.0:1787829968.742270:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787829968.742272:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829968.742279:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829968.742285:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829968.742296:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280953472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829968.742347:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829968/real 1787829968] req@ffff8ab3f043b840 x1874671280952960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829968.742364:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280952960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829968.742374:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829968.742378:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829968.742382:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280952960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829968.742389:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280953088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829968.742407:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829968.742419:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829968/real 1787829968] req@ffff8ab3f043b480 x1874671280953088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829968.742429:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280953088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829968.742438:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829968.742441:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829968.742444:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280953088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829968.742464:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280953216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829968.742479:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829968.742489:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829968/real 1787829968] req@ffff8ab3f0439e00 x1874671280953216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829968.742499:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280953216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829968.742513:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829968.742516:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829968.742519:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280953216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829968.742525:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280953344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829968.742540:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829968.742549:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829968/real 1787829968] req@ffff8ab3f043a1c0 x1874671280953344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829968.742559:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280953344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829968.742569:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829968.742571:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829968.742575:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280953344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829968.742580:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280953472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829968.742594:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829968.742602:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829968/real 1787829968] req@ffff8ab3f043a940 x1874671280953472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829968.742612:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280953472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829968.742633:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829968.742636:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829968.742639:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280953472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829968.742845:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829969.766140:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829970.790139:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829971.814146:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829972.838054:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829973.862041:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829973.862049:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829973.862059:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829973.862064:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829973.862073:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4523 00000100:00080000:0.0:1787829973.862082:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829973.862102:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829973.862140:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280953600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829973.862170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829973.862174:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829973.862176:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829973.862180:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4523 00000100:00080000:0.0:1787829973.862183:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829973.862188:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829973.862198:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829973.862212:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280953728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829973.862221:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829973.862225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829973.862227:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829973.862231:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4523 00000100:00080000:0.0:1787829973.862235:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829973.862238:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829973.862243:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829973.862297:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280953600:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787829973.862355:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829973.862415:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829973/real 1787829973] req@ffff8ab3f075c000 x1874671280953600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829973.862432:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280953600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829973.862442:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829973.862446:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829973.862450:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280953600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829973.862457:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280953728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829973.862490:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829973.862504:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829973/real 1787829973] req@ffff8ab3f075cb40 x1874671280953728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829973.862516:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280953728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829973.862524:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829973.862527:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829973.862531:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280953728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787829973.862586:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280953856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829973.862605:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829973.862608:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829973.862610:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829973.862614:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4523 00000100:00080000:0.0:1787829973.862618:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829973.862621:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829973.862627:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787829973.862636:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280953984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829973.862643:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829973.862646:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829973.862648:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829973.862652:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4523 00000100:00080000:0.0:1787829973.862655:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829973.862658:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829973.862662:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829973.862670:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280954112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829973.862796:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280953856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829973.862819:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829973.862832:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829973/real 1787829973] req@ffff8ab3f075cf00 x1874671280953856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829973.862860:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280953856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829973.862869:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829973.862871:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829973.862873:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280953856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829973.862878:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280953984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829973.862911:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829973.862920:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829973/real 1787829973] req@ffff8ab3f075e940 x1874671280953984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829973.862936:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280953984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829973.862945:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829973.862949:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829973.862951:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280953984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829973.862957:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280954112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829973.862975:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829973.862986:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829973/real 1787829973] req@ffff8ab3f075de00 x1874671280954112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829973.862998:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280954112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829973.863007:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829973.863010:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829973.863013:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280954112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787829974.886100:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829975.910181:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787829976.934653:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829977.958090:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829978.982019:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829978.982031:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829978.982037:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829978.982049:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4528 00000100:00080000:0.0:1787829978.982062:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829978.982071:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787829978.982121:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280954240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787829978.982123:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829978.982153:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829978.982157:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829978.982159:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829978.982163:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4528 00000100:00080000:0.0:1787829978.982167:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829978.982171:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829978.982184:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829978.982200:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280954240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787829978.982202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280954368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829978.982218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829978.982222:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829978.982224:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829978.982229:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4528 00000100:00080000:0.0:1787829978.982232:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829978.982237:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829978.982242:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829978.982255:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280954496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829978.982265:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829978.982269:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829978.982271:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829978.982275:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4528 00000100:00080000:0.0:1787829978.982280:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829978.982284:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829978.982290:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829978.982291:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787829978.982305:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280954624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829978.982315:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829978.982318:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829978.982320:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829978.982324:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4528 00000100:00080000:0.0:1787829978.982327:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829978.982330:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787829978.982335:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787829978.982347:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280954752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829978.982361:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829978/real 1787829978] req@ffff8ab3f075f0c0 x1874671280954240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829978.982381:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280954240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829978.982392:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829978.982396:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829978.982399:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280954240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829978.982440:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280954368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829978.982461:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829978.982473:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829978/real 1787829978] req@ffff8ab3f075f480 x1874671280954368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829978.982484:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280954368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829978.982495:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829978.982498:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829978.982501:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280954368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829978.982507:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280954496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829978.982525:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829978.982541:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829978/real 1787829978] req@ffff8ab3f075ed00 x1874671280954496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829978.982552:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280954496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829978.982561:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829978.982563:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829978.982567:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280954496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829978.982572:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280954624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829978.982713:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829978.982749:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829978/real 1787829978] req@ffff8ab3f075da40 x1874671280954624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829978.982768:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280954624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829978.982783:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829978.982787:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829978.982792:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280954624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829978.982803:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280954752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829978.982847:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829978.982858:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829978/real 1787829978] req@ffff8ab3f075f840 x1874671280954752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829978.982870:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280954752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829978.982880:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829978.982883:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829978.982933:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280954752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829980.007183:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829981.030096:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829982.054291:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829983.079552:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829984.102084:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829984.102095:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829984.102100:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829984.102109:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4533 00000100:00080000:3.0:1787829984.102120:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829984.102127:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829984.102135:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829984.102160:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280954880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829984.102185:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829984.102190:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829984.102192:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829984.102194:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4533 00000100:00080000:3.0:1787829984.102199:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829984.102202:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829984.102210:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829984.102221:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280955008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829984.102231:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280954880:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787829984.102249:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829984.102253:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829984.102255:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829984.102261:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4533 00000100:00080000:3.0:1787829984.102266:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829984.102270:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829984.102276:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829984.102286:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280955136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829984.102296:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829984.102299:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829984.102300:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829984.102304:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4533 00000100:00080000:3.0:1787829984.102307:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829984.102314:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829984.102316:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787829984.102321:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829984.102335:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280955264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829984.102344:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829984.102346:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829984.102348:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829984.102353:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4533 00000100:00080000:3.0:1787829984.102357:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829984.102361:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829984.102366:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787829984.102368:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829984/real 1787829984] req@ffff8ab3d0a421c0 x1874671280954880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829984.102380:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280955392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829984.102399:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280954880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829984.102411:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829984.102415:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829984.102419:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280954880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829984.102441:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280955008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829984.102474:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829984.102488:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829984/real 1787829984] req@ffff8ab3d0a40000 x1874671280955008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829984.102500:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280955008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829984.102509:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829984.102512:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829984.102515:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280955008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829984.102521:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280955136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829984.102538:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829984.102548:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829984/real 1787829984] req@ffff8ab3d0a43c00 x1874671280955136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829984.102568:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280955136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829984.102576:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829984.102580:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829984.102583:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280955136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829984.102589:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280955264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829984.102607:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829984.102617:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829984/real 1787829984] req@ffff8ab3d0a41e00 x1874671280955264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829984.102627:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280955264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829984.102637:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829984.102640:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829984.102643:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280955264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829984.102648:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280955392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829984.102665:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829984.102674:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829984/real 1787829984] req@ffff8ab3d0a430c0 x1874671280955392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829984.102684:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280955392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829984.102693:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829984.102697:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829984.102700:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280955392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829985.126145:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829986.153627:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829987.174354:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829988.198077:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829989.222015:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829989.222025:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829989.222029:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829989.222039:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4538 00000100:00080000:2.0:1787829989.222048:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829989.222055:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787829989.222090:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280955520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829989.222116:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829989.222119:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829989.222120:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829989.222124:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4538 00000100:00080000:2.0:1787829989.222127:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829989.222131:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829989.222140:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00010000:00010000:3.0:1787829989.222146:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829989.222154:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280955648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829989.222172:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829989.222175:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829989.222177:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829989.222184:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4538 00000100:00080000:2.0:1787829989.222187:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829989.222191:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829989.222196:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829989.222207:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280955776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829989.222216:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829989.222219:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829989.222221:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829989.222224:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4538 00000100:00080000:2.0:1787829989.222227:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829989.222231:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829989.222236:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787829989.222248:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280955520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787829989.222249:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280955904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829989.222264:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829989.222267:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829989.222269:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829989.222272:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4538 00000100:00080000:2.0:1787829989.222276:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829989.222279:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787829989.222284:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787829989.222294:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280956032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829989.222311:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829989.222358:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829989/real 1787829989] req@ffff8ab3f0439a40 x1874671280955520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829989.222372:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280955520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829989.222379:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829989.222382:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829989.222385:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280955520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829989.222390:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280955648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829989.222405:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829989.222413:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829989/real 1787829989] req@ffff8ab3f043bc00 x1874671280955648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829989.222421:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280955648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829989.222426:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829989.222427:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829989.222429:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280955648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829989.222432:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280955776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829989.222441:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829989.222446:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829989/real 1787829989] req@ffff8ab3f04383c0 x1874671280955776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829989.222453:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280955776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829989.222458:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829989.222461:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829989.222463:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280955776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829989.222486:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280955904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829989.222504:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829989.222515:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829989/real 1787829989] req@ffff8ab3f043ad00 x1874671280955904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829989.222526:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280955904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829989.222533:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829989.222534:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829989.222536:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280955904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829989.222540:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280956032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829989.222552:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829989.222560:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829989/real 1787829989] req@ffff8ab3f04392c0 x1874671280956032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829989.222571:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280956032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829989.222580:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829989.222583:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829989.222585:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280956032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787829990.246050:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829991.270158:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829992.294489:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829993.318064:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829994.341966:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829994.341976:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829994.341979:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829994.341989:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4543 00000100:00080000:3.0:1787829994.342000:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829994.342008:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787829994.342024:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829994.342044:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280956160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829994.342069:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829994.342073:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829994.342074:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829994.342078:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4543 00000100:00080000:3.0:1787829994.342081:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829994.342085:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829994.342094:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829994.342105:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280956288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829994.342115:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829994.342118:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829994.342120:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829994.342125:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4543 00000100:00080000:3.0:1787829994.342127:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829994.342135:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829994.342136:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280956160:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787829994.342140:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829994.342153:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280956416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829994.342162:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829994.342165:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829994.342167:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829994.342171:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4543 00000100:00080000:3.0:1787829994.342184:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829994.342189:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829994.342193:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787829994.342197:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787829994.342210:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280956544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829994.342220:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829994.342223:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829994.342225:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829994.342230:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4543 00000100:00080000:3.0:1787829994.342234:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829994.342241:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829994/real 1787829994] req@ffff8ab3d0a412c0 x1874671280956160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829994.342241:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829994.342247:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829994.342262:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280956672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829994.342268:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280956160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829994.342283:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829994.342287:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829994.342291:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280956160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829994.342296:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280956288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829994.342313:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829994.342331:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829994/real 1787829994] req@ffff8ab3d0a43840 x1874671280956288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829994.342346:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280956288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829994.342355:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829994.342358:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829994.342361:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280956288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829994.342385:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280956416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829994.342404:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829994.342415:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829994/real 1787829994] req@ffff8ab3d0a43480 x1874671280956416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829994.342427:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280956416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829994.342434:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829994.342437:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829994.342439:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280956416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829994.342443:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280956544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829994.342457:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829994.342464:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829994/real 1787829994] req@ffff8ab3d0a430c0 x1874671280956544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829994.342471:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280956544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829994.342477:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829994.342479:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829994.342481:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280956544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829994.342484:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280956672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829994.342513:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829994.342524:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829994/real 1787829994] req@ffff8ab3d0a41e00 x1874671280956672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829994.342534:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280956672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829994.342543:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829994.342546:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829994.342549:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280956672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787829995.366069:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829996.390016:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787829997.414042:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787829998.438079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829999.462057:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1787829999.462065:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829999.462072:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829999.462078:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829999.462088:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4549 00000100:00080000:3.0:1787829999.462098:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829999.462105:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787829999.462151:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280956800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829999.462180:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829999.462184:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829999.462187:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829999.462199:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4549 00000100:00080000:3.0:1787829999.462204:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829999.462209:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787829999.462221:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280956800:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787829999.462222:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829999.462242:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280956928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829999.462256:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829999.462259:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829999.462261:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829999.462266:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4549 00000100:00080000:3.0:1787829999.462279:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829999.462286:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787829999.462293:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787829999.462295:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829999.462308:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280957056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829999.462316:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829999.462319:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829999.462320:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829999.462323:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4549 00000100:00080000:3.0:1787829999.462325:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829999.462329:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829999.462335:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787829999.462349:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280957184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829999.462357:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829999.462358:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829999/real 1787829999] req@ffff8ab3d0a43c00 x1874671280956800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829999.462367:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829999.462372:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829999.462383:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4549 00000100:00080000:3.0:1787829999.462390:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829999.462395:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787829999.462400:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787829999.462409:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280957312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829999.462417:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280956800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829999.462431:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829999.462437:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829999.462441:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280956800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787829999.462460:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280956928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829999.462477:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829999.462486:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829999/real 1787829999] req@ffff8ab3d0a40000 x1874671280956928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829999.462494:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280956928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829999.462499:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829999.462501:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829999.462503:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280956928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829999.462507:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280957056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787829999.462516:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829999.462522:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829999/real 1787829999] req@ffff8ab3d0a421c0 x1874671280957056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829999.462528:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280957056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829999.462537:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829999.462539:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829999.462541:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280957056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787829999.462545:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280957184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829999.462553:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829999.462558:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829999/real 1787829999] req@ffff8ab3d0a41e00 x1874671280957184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829999.462564:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280957184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829999.462571:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829999.462572:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829999.462574:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280957184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787829999.462577:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280957312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787829999.462585:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787829999.462590:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829999/real 1787829999] req@ffff8ab3d0a430c0 x1874671280957312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829999.462602:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280957312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829999.462608:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829999.462610:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787829999.462612:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280957312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830000.486068:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830001.510117:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830002.534124:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830003.559013:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830004.582827:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830004.582838:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830004.582842:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830004.582852:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4554 00000100:00080000:0.0:1787830004.582862:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830004.582869:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830004.582929:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280957440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830004.582954:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830004.582958:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830004.582960:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830004.582964:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4554 00000100:00080000:0.0:1787830004.582967:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830004.582971:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830004.582980:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830004.582994:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280957568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830004.583003:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830004.583006:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830004.583008:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830004.583012:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4554 00000100:00080000:0.0:1787830004.583016:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830004.583020:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830004.583024:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830004.583035:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280957696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830004.583044:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830004.583047:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830004.583049:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830004.583053:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4554 00000100:00080000:0.0:1787830004.583056:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830004.583061:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830004.583066:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830004.583076:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280957824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830004.583085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830004.583088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830004.583090:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830004.583093:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4554 00000100:00080000:0.0:1787830004.583097:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830004.583100:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830004.583105:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830004.583116:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280957952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830004.584669:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280957440:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830004.584731:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830004.584787:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830004/real 1787830004] req@ffff8ab3f075e580 x1874671280957440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830004.584804:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280957440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830004.584815:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830004.584820:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830004.584823:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280957440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830004.584830:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280957568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830004.584852:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830004.584866:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830004/real 1787830004] req@ffff8ab3f075f840 x1874671280957568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830004.584876:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280957568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830004.584884:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830004.584908:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830004.584911:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280957568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830004.584917:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280957696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830004.584939:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830004.584951:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830004/real 1787830004] req@ffff8ab3f075da40 x1874671280957696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830004.584962:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280957696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830004.584969:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830004.584972:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830004.584975:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280957696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830004.584980:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280957824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830004.584999:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830004.585010:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830004/real 1787830004] req@ffff8ab3f075ed00 x1874671280957824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830004.585019:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280957824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830004.585028:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830004.585031:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830004.585034:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280957824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830004.585040:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280957952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830004.585057:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830004.585067:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830004/real 1787830004] req@ffff8ab3f075f480 x1874671280957952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830004.585077:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280957952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830004.585086:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830004.585089:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830004.585092:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280957952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830004.585780:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830005.606113:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830006.630052:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830007.656707:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830008.678131:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830009.702095:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830009.702106:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830009.702111:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830009.702125:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4559 00000100:00080000:0.0:1787830009.702138:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830009.702146:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:3.0:1787830009.702186:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830009.702197:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280958080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830009.702415:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280958080:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830009.702493:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830009.702846:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830009.702856:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830009.702860:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830009.702870:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4559 00000100:00080000:0.0:1787830009.702881:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830009.702915:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830009/real 1787830009] req@ffff8ab3f075f0c0 x1874671280958080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830009.702939:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830009.702940:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280958080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787830009.702957:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787830009.702967:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830009.702976:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830009.702981:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280958080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830009.703011:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280958208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830009.703046:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830009.703054:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830009.703056:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830009.703060:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4559 00000100:00080000:0.0:1787830009.703064:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830009.703068:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830009.703072:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830009.703090:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280958336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830009.703100:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830009.703103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830009.703104:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830009.703108:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4559 00000100:00080000:0.0:1787830009.703111:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830009.703115:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830009.703119:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830009.703127:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280958464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830009.703134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830009.703137:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830009.703139:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830009.703142:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4559 00000100:00080000:0.0:1787830009.703144:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830009.703147:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830009.703151:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830009.703158:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280958592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830009.703393:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280958208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830009.703422:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830009.703436:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830009/real 1787830009] req@ffff8ab3f075de00 x1874671280958208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830009.703448:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280958208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830009.703457:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830009.703461:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830009.703464:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280958208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830009.703657:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280958336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830009.703680:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830009.703693:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830009/real 1787830009] req@ffff8ab3f075e940 x1874671280958336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830009.703705:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280958336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830009.703714:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830009.703718:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830009.703722:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280958336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830009.703728:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280958464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830009.703746:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830009.703757:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830009/real 1787830009] req@ffff8ab3f075cf00 x1874671280958464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830009.703768:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280958464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830009.703846:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830009.703851:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830009.703854:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280958464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830009.703861:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280958592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830009.703883:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830009.703981:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830009/real 1787830009] req@ffff8ab3f075cb40 x1874671280958592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830009.703995:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280958592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830009.704005:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830009.704009:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830009.704012:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280958592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830010.726620:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830011.750155:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830012.774199:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830013.798120:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830014.822165:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830014.822279:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830014.822285:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830014.822290:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830014.822298:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4564 00000100:00080000:0.0:1787830014.822307:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830014.822314:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830014.822348:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280958720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830014.822381:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830014.822387:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830014.822388:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830014.822392:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4564 00000100:00080000:0.0:1787830014.822395:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830014.822399:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830014.822407:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830014.822420:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280958848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830014.822429:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830014.822432:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830014.822434:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830014.822439:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4564 00000100:00080000:0.0:1787830014.822442:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830014.822446:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830014.822451:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830014.822459:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280958720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830014.822462:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280958976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830014.822474:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830014.822476:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830014.822478:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830014.822481:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4564 00000100:00080000:0.0:1787830014.822485:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830014.822489:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830014.822493:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830014.822503:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280959104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830014.822510:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830014.822513:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830014.822514:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830014.822518:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4564 00000100:00080000:0.0:1787830014.822521:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830014.822524:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830014.822527:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830014.822536:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280959232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830014.822584:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280958848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830014.822621:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280958976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830014.822643:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280959104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830014.822663:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280959232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830015.846116:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830016.870035:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787830017.895012:0:10942:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787830017.895054:0:10942:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787830017.895070:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787830017.895110:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787830017.895119:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787830017.895127:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787830017.895135:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787830017.895208:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830014/real 1787830017] req@ffff8ab3f075cf00 x1874671280958976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830017.895231:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280958976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830017.895243:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830017.895247:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830017.895252:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280958976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830017.895259:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830014/real 1787830017] req@ffff8ab3f075e940 x1874671280959104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830017.895278:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280959104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830017.895287:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830017.895290:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830017.895293:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280959104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830017.895299:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830014/real 1787830017] req@ffff8ab3f075de00 x1874671280959232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830017.895310:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280959232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830017.895320:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830017.895323:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830017.895326:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280959232:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830017.895332:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830014/real 1787830017] req@ffff8ab3f075c000 x1874671280958720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830017.895342:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280958720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830017.895349:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830017.895352:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830017.895355:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280958720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830017.895361:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830014/real 1787830017] req@ffff8ab3f075cb40 x1874671280958848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830017.895371:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280958848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830017.895378:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830017.895381:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830017.895384:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280958848:192.168.203.124@tcp:38:kworker.0 00010000:00010000:3.0:1787830017.895553:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830018.919158:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830019.942344:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830019.942466:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830019.942473:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830019.942477:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830019.942486:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4569 00000100:00080000:2.0:1787830019.942495:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830019.942502:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830019.942545:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280959360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830019.942606:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280959360:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830019.942658:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830019.942686:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830019.942690:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830019.942691:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830019.942695:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4569 00000100:00080000:2.0:1787830019.942699:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830019.942703:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830019.942712:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830019.942724:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280959488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830019.942821:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830019/real 1787830019] req@ffff8ab3f0439680 x1874671280959360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830019.942838:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280959360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830019.942848:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830019.942852:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830019.942855:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280959360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830019.942876:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280959488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830019.942915:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830019.942927:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830019/real 1787830019] req@ffff8ab3f043b0c0 x1874671280959488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830019.942938:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280959488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830019.942952:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830019.942955:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830019.942958:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280959488:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787830019.945049:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830019.945063:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830019.945067:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830019.945076:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4569 00000100:00080000:2.0:1787830019.945085:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830019.945091:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830019.945103:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830019.945130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280959616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830019.945152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830019.945156:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830019.945158:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830019.945162:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4569 00000100:00080000:2.0:1787830019.945174:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830019.945179:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830019.945184:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830019.945193:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280959616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830019.945198:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280959744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830019.945208:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830019.945211:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830019.945213:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830019.945216:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4569 00000100:00080000:2.0:1787830019.945219:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830019.945224:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830019.945238:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787830019.945243:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830019.945251:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280959872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830019.945269:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830019/real 1787830019] req@ffff8ab3f043a580 x1874671280959616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830019.945285:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280959616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830019.945295:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830019.945298:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830019.945302:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280959616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830019.945318:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280959744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830019.945335:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830019.945344:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830019/real 1787830019] req@ffff8ab3f0438f00 x1874671280959744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830019.945354:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280959744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830019.945363:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830019.945366:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830019.945369:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280959744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830019.945374:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280959872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830019.945389:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830019.945397:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830019/real 1787830019] req@ffff8ab3f0438b40 x1874671280959872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830019.945406:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280959872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830019.945415:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830019.945418:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830019.945421:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280959872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830020.966510:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830021.990780:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830023.014089:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830024.038120:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830025.062041:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830025.062055:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830025.062067:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830025.062083:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4574 00000100:00080000:2.0:1787830025.062094:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830025.062103:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830025.062160:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280960000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830025.062188:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830025.062192:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830025.062194:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830025.062198:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4574 00000100:00080000:2.0:1787830025.062202:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830025.062207:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830025.062219:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830025.062336:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280960000:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830025.062408:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830025.062481:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830025/real 1787830025] req@ffff8ab3f0438780 x1874671280960000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830025.062511:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280960000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830025.062523:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830025.062527:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830025.062531:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280960000:192.168.203.124@tcp:250:kworker.0 00010000:00010000:3.0:1787830025.062646:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830025.063332:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280960128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830025.063357:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830025.063361:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830025.063364:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830025.063368:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4574 00000100:00080000:2.0:1787830025.063373:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830025.063378:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830025.063385:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830025.063398:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280960256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830025.063407:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280960128:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787830025.063414:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830025.063418:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830025.063421:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830025.063427:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4574 00000100:00080000:2.0:1787830025.063438:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787830025.063442:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830025.063448:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830025.063455:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830025.063463:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830025/real 1787830025] req@ffff8ab3f0438780 x1874671280960128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830025.063470:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280960384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830025.063484:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280960128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830025.063485:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830025.063502:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830025.063506:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830025.063514:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830025.063514:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4574 00000100:00080000:2.0:1787830025.063522:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830025.063524:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787830025.063530:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830025.063532:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280960128:192.168.203.124@tcp:38:kworker.0 00000008:00100000:2.0:1787830025.063537:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830025.063549:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280960256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830025.063553:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280960512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830025.063588:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830025.063600:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830025/real 1787830025] req@ffff8ab3f0438b40 x1874671280960256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830025.063612:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280960256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830025.063620:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830025.063623:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830025.063626:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280960256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830025.063632:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280960384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830025.063651:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830025.063661:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830025/real 1787830025] req@ffff8ab3f0438f00 x1874671280960384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830025.063671:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280960384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830025.063682:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830025.063685:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830025.063689:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280960384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830025.063700:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280960512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830025.063717:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830025.063726:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830025/real 1787830025] req@ffff8ab3f043a580 x1874671280960512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830025.063737:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280960512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830025.063747:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830025.063750:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830025.063753:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280960512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830026.086313:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830027.110112:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830028.134390:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830029.158422:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830030.182019:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830030.182027:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830030.182031:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830030.182040:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4579 00000100:00080000:3.0:1787830030.182047:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830030.182053:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830030.182091:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280960640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830030.182106:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830030.182110:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830030.182112:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830030.182117:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4579 00000100:00080000:3.0:1787830030.182120:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830030.182124:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787830030.182127:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830030.182133:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280960640:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787830030.182135:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830030.182152:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280960768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830030.182161:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830030.182163:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830030.182164:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830030.182167:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4579 00000800:00000100:1.0:1787830030.182254:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830030.182356:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830030.182362:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830030.182366:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830030.182376:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280960896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830030.182383:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830030.182385:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830030.182386:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830030.182388:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4579 00000100:00080000:3.0:1787830030.182428:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830030.182431:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830030.182434:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830030.182444:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280961024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830030.182463:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830030.182467:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830030.182471:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830030.182473:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830030/real 1787830030] req@ffff8ab3d0a43480 x1874671280960640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830030.182478:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4579 00000100:00080000:3.0:1787830030.182484:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830030.182491:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830030.182497:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830030.182500:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280960640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830030.182511:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280961152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830030.182512:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830030.182520:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830030.182524:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280960640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830030.182547:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280960768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830030.182569:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830030.182580:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830030/real 1787830030] req@ffff8ab3d0a43840 x1874671280960768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830030.182592:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280960768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830030.182600:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830030.182603:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830030.182606:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280960768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830030.182612:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280960896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830030.182625:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830030.182634:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830030/real 1787830030] req@ffff8ab3d0a412c0 x1874671280960896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830030.182644:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280960896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830030.182652:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830030.182655:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830030.182658:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280960896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830030.182662:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280961024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830030.182688:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830030.182698:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830030/real 1787830030] req@ffff8ab3d0a430c0 x1874671280961024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830030.182708:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280961024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830030.182717:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830030.182720:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830030.182723:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280961024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830030.182729:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280961152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830030.182742:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830030.182750:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830030/real 1787830030] req@ffff8ab3d0a41e00 x1874671280961152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830030.182760:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280961152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830030.182768:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830030.182771:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830030.182774:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280961152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830031.206156:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830032.230400:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830033.254175:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830034.278282:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830035.302019:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830035.302027:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830035.302033:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830035.302041:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4584 00000100:00080000:3.0:1787830035.302050:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830035.302055:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830035.302088:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280961280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830035.302109:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830035.302112:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830035.302114:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830035.302116:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4584 00000100:00080000:3.0:1787830035.302118:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830035.302121:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830035.302128:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830035.302136:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280961408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830035.302142:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830035.302144:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830035.302145:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830035.302147:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4584 00000100:00080000:3.0:1787830035.302149:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830035.302151:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830035.302153:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830035.302160:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280961536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830035.302166:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830035.302169:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830035.302170:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830035.302173:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4584 00000100:00080000:3.0:1787830035.302177:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1787830035.302180:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830035.302180:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830035.302186:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830035.302198:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280961664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830035.302204:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830035.302206:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830035.302207:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830035.302209:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4584 00000100:00080000:3.0:1787830035.302210:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830035.302212:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830035.302214:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830035.302219:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280961792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830035.302269:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280961280:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830035.302348:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830035.302406:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830035/real 1787830035] req@ffff8ab3d0a421c0 x1874671280961280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830035.302444:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280961280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830035.302457:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830035.302462:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830035.302467:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280961280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830035.302475:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280961408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830035.302498:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830035.302511:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830035/real 1787830035] req@ffff8ab3d0a40000 x1874671280961408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830035.302524:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280961408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830035.302546:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830035.302550:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830035.302553:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280961408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830035.302561:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280961536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830035.302581:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830035.302592:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830035/real 1787830035] req@ffff8ab3d0a43c00 x1874671280961536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830035.302604:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280961536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830035.302611:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830035.302614:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830035.302617:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280961536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830035.302624:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280961664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830035.302644:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830035.302656:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830035/real 1787830035] req@ffff8ab3d0a41e00 x1874671280961664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830035.302667:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280961664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830035.302677:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830035.302681:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830035.302684:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280961664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830035.302691:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280961792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830035.302708:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830035.302719:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830035/real 1787830035] req@ffff8ab3d0a430c0 x1874671280961792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830035.302731:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280961792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830035.302740:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830035.302743:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830035.302746:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280961792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830036.326425:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830037.350298:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830038.374254:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830039.409315:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830040.422232:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830040.423187:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830040.423197:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830040.423201:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830040.423212:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4590 00000100:00080000:0.0:1787830040.423224:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830040.423232:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830040.423282:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280961920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830040.423372:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830040.423376:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830040.423378:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830040.423432:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280961920:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830040.423518:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830040.423570:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830040/real 1787830040] req@ffff8ab3f075f0c0 x1874671280961920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830040.423589:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280961920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830040.423600:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830040.423605:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830040.423609:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280961920:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830040.423666:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4590 00000100:00080000:0.0:1787830040.423670:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830040.423674:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830040.423686:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830040.423699:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280962048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830040.423717:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830040.423721:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830040.423723:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830040.423727:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4590 00000100:00080000:0.0:1787830040.423730:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830040.423735:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830040.423739:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830040.423750:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280962176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830040.423759:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830040.423762:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830040.423764:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830040.423768:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4590 00000100:00080000:0.0:1787830040.423771:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830040.423775:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830040.423779:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830040.423788:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280962304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830040.423797:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830040.423800:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830040.423802:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830040.423806:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4590 00000100:00080000:0.0:1787830040.423809:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830040.423813:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830040.423817:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830040.423826:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280962432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830040.423941:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280962048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830040.423966:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830040.423993:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830040/real 1787830040] req@ffff8ab3f075f480 x1874671280962048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830040.424006:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280962048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830040.424015:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830040.424018:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830040.424021:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280962048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830040.424027:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280962176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830040.424045:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830040.424055:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830040/real 1787830040] req@ffff8ab3f075ed00 x1874671280962176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830040.424066:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280962176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830040.424080:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830040.424083:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830040.424086:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280962176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830040.424091:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280962304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830040.424108:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830040.424117:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830040/real 1787830040] req@ffff8ab3f075da40 x1874671280962304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830040.424128:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280962304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830040.424138:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830040.424141:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830040.424144:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280962304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830040.424149:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280962432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830040.424166:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830040.424175:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830040/real 1787830040] req@ffff8ab3f075f840 x1874671280962432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830040.424186:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280962432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830040.424195:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830040.424198:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830040.424201:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280962432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830041.446785:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830042.470357:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830043.494721:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830044.518285:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830045.542107:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830045.542117:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830045.542122:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830045.542131:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4595 00000100:00080000:3.0:1787830045.542141:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830045.542147:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830045.542182:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280962560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830045.542239:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280962560:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830045.542298:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830045.542316:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830045.542320:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830045.542322:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830045.542326:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4595 00000100:00080000:3.0:1787830045.542330:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830045.542334:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830045.542355:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830045.542369:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280962688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830045.542380:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830045.542383:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830045.542385:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830045.542389:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4595 00000100:00080000:3.0:1787830045.542393:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830045.542397:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787830045.542398:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:3.0:1787830045.542403:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830045.542414:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280962816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830045.542422:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830045.542425:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830045.542427:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830045.542430:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4595 00000100:00080000:3.0:1787830045.542433:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830045.542436:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830045.542440:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830045.542450:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280962944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830045.542457:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830045.542460:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830045.542462:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830045.542466:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4595 00000100:00080000:3.0:1787830045.542468:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830045.542472:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830045.542475:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830045.542482:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280963072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830045.542557:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830045/real 1787830045] req@ffff8ab3d0a412c0 x1874671280962560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830045.542572:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280962560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830045.542581:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830045.542584:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830045.542588:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280962560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830045.542608:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280962688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830045.542627:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830045.542638:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830045/real 1787830045] req@ffff8ab3d0a43840 x1874671280962688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830045.542649:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280962688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830045.542657:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830045.542659:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830045.542662:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280962688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830045.542668:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280962816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830045.542681:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830045.542690:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830045/real 1787830045] req@ffff8ab3d0a43480 x1874671280962816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830045.542701:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280962816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830045.542709:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830045.542712:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830045.542715:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280962816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830045.542720:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280962944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830045.542734:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830045.542742:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830045/real 1787830045] req@ffff8ab3d0a430c0 x1874671280962944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830045.542752:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280962944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830045.542761:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830045.542764:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830045.542767:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280962944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830045.542773:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280963072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830045.542787:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830045.542795:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830045/real 1787830045] req@ffff8ab3d0a41e00 x1874671280963072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830045.542805:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280963072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830045.542815:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830045.542818:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830045.542821:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280963072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830046.567555:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830047.590322:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830048.614113:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830049.638212:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830050.662235:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830050.662245:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830050.662251:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830050.662260:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4600 00000100:00080000:2.0:1787830050.662271:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830050.662279:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830050.662336:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280963200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830050.662361:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1787830050.662362:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830050.662367:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830050.662369:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830050.662374:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4600 00000100:00080000:2.0:1787830050.662378:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830050.662383:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830050.662391:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830050.662402:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280963328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830050.662410:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830050.662413:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830050.662414:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830050.662418:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4600 00000100:00080000:2.0:1787830050.662420:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830050.662423:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830050.662427:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830050.662436:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280963456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830050.662449:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830050.662452:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830050.662453:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830050.662456:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4600 00000100:00080000:2.0:1787830050.662460:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830050.662464:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830050.662469:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830050.662479:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280963584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830050.662484:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280963200:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787830050.662492:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830050.662497:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830050.662498:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830050.662502:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4600 00000100:00080000:2.0:1787830050.662505:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830050.662508:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830050.662512:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830050.662522:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280963712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830050.662552:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830050.662619:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830050/real 1787830050] req@ffff8ab3f043b0c0 x1874671280963200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830050.662635:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280963200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830050.662645:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830050.662649:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830050.662653:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280963200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830050.662661:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280963328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830050.662682:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830050.662695:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830050/real 1787830050] req@ffff8ab3f0439680 x1874671280963328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830050.662707:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280963328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830050.662716:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830050.662719:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830050.662722:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280963328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830050.662729:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280963456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830050.662746:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830050.662768:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830050/real 1787830050] req@ffff8ab3f04392c0 x1874671280963456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830050.662780:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280963456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830050.662789:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830050.662792:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830050.662795:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280963456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830050.662818:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280963584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830050.662837:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830050.662848:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830050/real 1787830050] req@ffff8ab3f043ad00 x1874671280963584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830050.662859:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280963584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830050.662870:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830050.662875:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830050.662879:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280963584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830050.662906:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280963712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830050.662924:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830050.662944:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830050/real 1787830050] req@ffff8ab3f04383c0 x1874671280963712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830050.662956:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280963712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830050.662966:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830050.662969:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830050.662972:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280963712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830051.686123:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830052.710349:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830053.734131:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830054.758072:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830055.782083:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830055.782094:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830055.782099:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830055.782117:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4605 00000100:00080000:2.0:1787830055.782128:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830055.782135:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830055.782172:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280963840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830055.782209:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830055.782212:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830055.782214:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830055.782218:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4605 00000100:00080000:2.0:1787830055.782221:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830055.782226:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830055.782236:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830055.782253:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280963968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830055.782270:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280963840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787830055.782286:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830055.782291:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830055.782293:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830055.782302:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4605 00000100:00080000:2.0:1787830055.782307:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830055.782313:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830055.782319:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830055.782334:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280964096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830055.782350:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830055.782353:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830055.782356:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787830055.782362:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830055.782371:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4605 00000100:00080000:2.0:1787830055.782378:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830055.782385:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830055.782391:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830055.782405:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280964224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830055.782413:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830055.782417:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830055.782419:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830055.782426:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4605 00000100:00080000:2.0:1787830055.782430:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830055.782434:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830055/real 1787830055] req@ffff8ab3f043bc00 x1874671280963840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830055.782438:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830055.782458:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830055.782466:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280963840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830055.782477:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280964352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830055.782477:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830055.782489:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830055.782499:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280963840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830055.782534:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280963968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830055.782556:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830055.782570:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830055/real 1787830055] req@ffff8ab3f0439a40 x1874671280963968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830055.782581:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280963968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830055.782589:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830055.782592:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830055.782596:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280963968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830055.782603:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280964096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830055.782622:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830055.782632:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830055/real 1787830055] req@ffff8ab3f043a940 x1874671280964096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830055.782643:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280964096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830055.782650:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830055.782654:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830055.782657:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280964096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830055.782664:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280964224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830055.782684:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830055.782696:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830055/real 1787830055] req@ffff8ab3f043a1c0 x1874671280964224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830055.782705:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280964224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830055.782713:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830055.782715:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830055.782718:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280964224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830055.782723:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280964352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830055.782739:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830055.782747:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830055/real 1787830055] req@ffff8ab3f0439e00 x1874671280964352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830055.782755:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280964352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830055.782762:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830055.782764:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830055.782766:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280964352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830055.782797:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830056.806407:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830057.830164:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830058.854182:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830059.878316:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830060.902122:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830060.902132:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830060.902136:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830060.902146:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4610 00000100:00080000:0.0:1787830060.902156:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830060.902163:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830060.902197:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280964480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830060.902228:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830060.902231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830060.902233:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830060.902245:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4610 00000100:00080000:0.0:1787830060.902249:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830060.902253:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830060.902262:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830060.902281:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280964608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830060.902291:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830060.902294:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830060.902296:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830060.902300:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4610 00000100:00080000:0.0:1787830060.902303:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830060.902307:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830060.902312:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830060.902322:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280964736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830060.902331:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830060.902333:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830060.902335:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830060.902339:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4610 00000100:00080000:0.0:1787830060.902341:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830060.902346:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830060.902350:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830060.902360:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280964864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830060.902368:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830060.902371:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830060.902372:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830060.902376:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4610 00000100:00080000:0.0:1787830060.902378:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830060.902381:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830060.902386:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830060.902394:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280964992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830060.902496:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280964480:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830060.902552:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830060.902597:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830060/real 1787830060] req@ffff8ab3f075e580 x1874671280964480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830060.902613:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280964480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830060.902627:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830060.902631:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830060.902635:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280964480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830060.902645:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280964608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830060.902684:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830060.902710:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830060/real 1787830060] req@ffff8ab3f075f840 x1874671280964608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830060.902727:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280964608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830060.902742:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830060.902748:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830060.902754:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280964608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830060.902763:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280964736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830060.902797:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830060.902814:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830060/real 1787830060] req@ffff8ab3f075da40 x1874671280964736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830060.902834:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280964736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830060.902844:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830060.902850:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830060.902855:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280964736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830060.902865:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280964864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830060.902915:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830060.902927:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830060/real 1787830060] req@ffff8ab3f075ed00 x1874671280964864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830060.902952:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280964864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830060.902971:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830060.902978:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830060.902984:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280964864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830060.902993:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280964992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830060.903028:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830060.903049:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830060/real 1787830060] req@ffff8ab3f075f480 x1874671280964992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830060.903070:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280964992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830060.903085:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830060.903090:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830060.903094:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280964992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830060.903367:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830061.927287:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830062.950558:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830063.977322:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830064.998615:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830066.022180:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830066.022194:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830066.022199:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830066.022220:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4615 00000100:00080000:2.0:1787830066.022233:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830066.022242:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830066.022290:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280965120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830066.022322:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830066.022325:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830066.022327:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830066.022331:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4615 00000100:00080000:2.0:1787830066.022335:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830066.022338:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830066.022349:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830066.022362:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280965248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830066.022371:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830066.022375:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830066.022376:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830066.022380:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4615 00000100:00080000:2.0:1787830066.022383:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830066.022387:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830066.022392:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830066.022418:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280965376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830066.022427:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830066.022431:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830066.022432:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830066.022437:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4615 00000100:00080000:2.0:1787830066.022440:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830066.022445:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830066.022449:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830066.022460:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280965504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830066.022479:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830066.022482:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830066.022649:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830066.022655:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4615 00000100:00080000:2.0:1787830066.022659:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830066.022664:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830066.022669:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830066.022682:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280965632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830066.023973:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280965120:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830066.024047:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830066.024099:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830066/real 1787830066] req@ffff8ab3f043b480 x1874671280965120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830066.024117:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280965120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830066.024129:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830066.024134:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830066.024138:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280965120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830066.024148:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280965248:192.168.203.124@tcp:38:kworker.0 00010000:00010000:2.0:1787830066.024155:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:1.0:1787830066.024172:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830066.024186:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830066/real 1787830066] req@ffff8ab3f043b840 x1874671280965248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830066.024211:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280965248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830066.024226:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830066.024230:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830066.024234:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280965248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830066.024241:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280965376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830066.024261:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830066.024271:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830066/real 1787830066] req@ffff8ab3f0439e00 x1874671280965376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830066.024282:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280965376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830066.024289:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830066.024293:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830066.024296:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280965376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830066.024302:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280965504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830066.024320:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830066.024330:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830066/real 1787830066] req@ffff8ab3f043a1c0 x1874671280965504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830066.024341:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280965504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830066.024351:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830066.024355:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830066.024359:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280965504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830066.024364:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280965632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830066.024383:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830066.024393:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830066/real 1787830066] req@ffff8ab3f043a940 x1874671280965632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830066.024403:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280965632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830066.024412:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830066.024415:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830066.024417:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280965632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830067.048559:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830068.070241:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830069.095449:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830070.118078:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830071.142215:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830071.142938:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830071.142948:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830071.142953:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830071.142982:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4620 00000100:00080000:3.0:1787830071.143001:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830071.143011:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830071.143078:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280965760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830071.143147:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830071.143152:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830071.143154:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830071.143182:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4620 00000100:00080000:3.0:1787830071.143186:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830071.143191:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830071.143203:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830071.143229:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280965888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830071.143289:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830071.143293:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830071.143306:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830071.143311:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4620 00000100:00080000:3.0:1787830071.143314:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830071.143319:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830071.143324:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830071.143355:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280966016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830071.143366:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830071.143370:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830071.143372:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830071.143388:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4620 00000100:00080000:3.0:1787830071.143392:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830071.143397:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830071.143403:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830071.143433:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280966144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830071.143442:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830071.143445:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830071.143447:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830071.143451:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4620 00000100:00080000:3.0:1787830071.143454:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830071.143484:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830071.143490:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830071.143521:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280966272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830071.143673:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280965760:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830071.143853:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830071.143991:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830071/real 1787830071] req@ffff8ab3d0a43c00 x1874671280965760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830071.144044:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280965760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830071.144076:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830071.144080:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830071.144085:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280965760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830071.144092:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280965888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830071.144116:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830071.144131:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830071/real 1787830071] req@ffff8ab3d0a40000 x1874671280965888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830071.144152:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280965888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830071.144161:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830071.144192:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830071.144196:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280965888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830071.144202:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280966016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830071.144233:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830071.144290:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830071/real 1787830071] req@ffff8ab3d0a421c0 x1874671280966016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830071.144304:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280966016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830071.144312:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830071.144315:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830071.144335:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280966016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830071.144343:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280966144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830071.144373:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830071.144406:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830071/real 1787830071] req@ffff8ab3d0a41e00 x1874671280966144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830071.144418:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280966144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830071.144435:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830071.144439:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830071.144442:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280966144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830071.144448:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280966272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830071.144477:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830071.144497:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830071/real 1787830071] req@ffff8ab3d0a430c0 x1874671280966272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830071.144508:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280966272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830071.144525:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830071.144544:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830071.144548:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280966272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830072.166400:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830073.190552:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830074.214392:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830075.238124:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830076.262043:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830076.262056:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830076.262062:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830076.262074:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4625 00000100:00080000:3.0:1787830076.262085:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830076.262093:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787830076.262138:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830076.262154:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280966400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830076.262184:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830076.262188:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830076.262190:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830076.262193:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4625 00000100:00080000:3.0:1787830076.262198:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830076.262202:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830076.262213:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830076.262227:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280966528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830076.262236:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830076.262240:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830076.262242:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830076.262246:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4625 00000100:00080000:3.0:1787830076.262249:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830076.262254:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830076.262258:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830076.262270:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280966656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830076.262273:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280966400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787830076.262279:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830076.262282:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830076.262284:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830076.262288:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4625 00000100:00080000:3.0:1787830076.262291:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830076.262295:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830076.262299:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830076.262310:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280966784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830076.262323:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830076.262326:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830076.262327:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830076.262331:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4625 00000100:00080000:3.0:1787830076.262334:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830076.262337:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830076.262341:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787830076.262344:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830076.262350:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280966912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830076.262398:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830076/real 1787830076] req@ffff8ab3d0a43480 x1874671280966400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830076.262417:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280966400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830076.262428:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830076.262432:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830076.262436:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280966400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830076.262444:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280966528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830076.262468:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830076.262483:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830076/real 1787830076] req@ffff8ab3d0a43840 x1874671280966528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830076.262495:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280966528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830076.262504:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830076.262507:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830076.262510:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280966528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830076.262549:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280966656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830076.262571:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830076.262584:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830076/real 1787830076] req@ffff8ab3d0a412c0 x1874671280966656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830076.262596:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280966656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830076.262604:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830076.262607:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830076.262611:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280966656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830076.262616:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280966784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830076.262637:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830076.262660:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830076/real 1787830076] req@ffff8ab3d0a430c0 x1874671280966784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830076.262672:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280966784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830076.262683:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830076.262686:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830076.262689:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280966784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830076.262708:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280966912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830076.262728:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830076.262739:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830076/real 1787830076] req@ffff8ab3d0a41e00 x1874671280966912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830076.262764:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280966912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830076.262774:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830076.262778:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830076.262781:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280966912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830077.287219:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830078.311646:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830079.334163:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830080.361603:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830081.382173:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830081.384632:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830081.384645:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830081.384650:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830081.384663:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4631 00000100:00080000:0.0:1787830081.384676:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830081.384685:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830081.384733:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280967040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830081.384881:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280967040:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830081.385048:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830081.385052:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830081.385054:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830081.385059:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4631 00000100:00080000:0.0:1787830081.385063:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830081.385068:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830081.385079:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830081.385450:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280967168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830081.397164:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830081.397176:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830081.397180:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830081.397206:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4631 00000100:00080000:0.0:1787830081.397218:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830081.397229:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830081.397253:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830081.397299:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280967296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830081.397317:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830081.397320:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830081.397322:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830081.397326:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4631 00000100:00080000:0.0:1787830081.397330:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830081.397334:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830081.397340:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830081.397351:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280967424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830081.397367:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830081.397370:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830081.397372:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830081.397376:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4631 00000100:00080000:0.0:1787830081.397380:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830081.397383:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830081.397388:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830081.397399:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280967552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830081.399563:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280967168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830081.399670:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280967296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830081.399696:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280967424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830081.399720:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280967552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830082.412210:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830083.431960:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830084.454221:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787830084.456383:0:10944:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:3.0:1787830084.456440:0:10944:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787830084.456459:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830084.456506:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787830084.456612:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830081/real 1787830084] req@ffff8ab3f075f0c0 x1874671280967040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830084.456640:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280967040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830084.456653:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830084.456658:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830084.456674:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280967040:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830084.459057:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787830084.459109:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830081/real 1787830084] req@ffff8ab3f075cb40 x1874671280967168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830084.459136:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280967168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830084.459149:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830084.459153:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830084.459157:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280967168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830084.459949:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830084.459964:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:1.0:1787830084.460146:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830081/real 1787830084] req@ffff8ab3f075c000 x1874671280967296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830084.460172:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280967296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830084.460183:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830084.460187:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830084.460192:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280967296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830084.460199:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830081/real 1787830084] req@ffff8ab3f075de00 x1874671280967424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830084.460211:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280967424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830084.460220:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830084.460224:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830084.460227:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280967424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830084.460233:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830081/real 1787830084] req@ffff8ab3f075e940 x1874671280967552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830084.460243:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280967552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830084.460253:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830084.460256:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830084.460259:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280967552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830085.483483:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830086.502018:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830086.502031:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830086.502036:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830086.502049:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4636 00000100:00080000:0.0:1787830086.502261:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830086.502272:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830086.502317:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280967680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830086.502354:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830086.502358:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830086.502361:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830086.502365:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4636 00000100:00080000:0.0:1787830086.502369:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830086.502374:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830086.502385:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830086.502399:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280967808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830086.502408:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830086.502412:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830086.502413:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830086.502417:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4636 00000100:00080000:0.0:1787830086.502420:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830086.502424:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830086.502429:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830086.502440:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280967936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830086.502449:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830086.502452:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830086.502454:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830086.502458:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4636 00000100:00080000:0.0:1787830086.502461:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830086.502465:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830086.502470:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830086.502481:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280968064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830086.502490:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830086.502493:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830086.502495:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830086.502499:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4636 00000100:00080000:0.0:1787830086.502502:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830086.502506:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830086.502510:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830086.502521:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280968192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830086.502640:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280967680:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830086.502716:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830086.502783:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830086/real 1787830086] req@ffff8ab3f075cf00 x1874671280967680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830086.502801:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280967680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830086.502820:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830086.502825:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830086.502829:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280967680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830086.502843:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280967808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830086.502867:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830086.502881:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830086/real 1787830086] req@ffff8ab3f075e940 x1874671280967808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830086.502919:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280967808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830086.502927:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830086.502931:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830086.502935:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280967808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830086.502941:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280967936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830086.502962:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830086.502972:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830086/real 1787830086] req@ffff8ab3f075de00 x1874671280967936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830086.502982:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280967936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830086.502990:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830086.502993:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830086.502996:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280967936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830086.503002:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280968064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830086.503021:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830086.503032:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830086/real 1787830086] req@ffff8ab3f075c000 x1874671280968064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830086.503043:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280968064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830086.503052:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830086.503055:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830086.503068:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280968064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830086.503074:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280968192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830086.503095:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830086.503105:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830086/real 1787830086] req@ffff8ab3f075cb40 x1874671280968192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830086.503114:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280968192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830086.503123:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830086.503126:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830086.503129:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280968192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830086.504095:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830087.528114:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830088.550397:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830089.574153:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830090.603223:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830091.622374:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830091.622449:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830091.622457:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830091.622462:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830091.622474:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4641 00000100:00080000:3.0:1787830091.622486:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830091.622495:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830091.622544:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280968320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830091.622576:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830091.622580:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830091.622582:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830091.622586:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4641 00000100:00080000:3.0:1787830091.622589:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830091.622594:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830091.622605:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830091.622617:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280968448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830091.622625:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830091.622629:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830091.622630:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830091.622634:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4641 00000100:00080000:3.0:1787830091.622637:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830091.622641:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830091.622646:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830091.622656:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280968576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830091.622672:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830091.622675:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830091.622677:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830091.622682:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4641 00000100:00080000:3.0:1787830091.622685:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830091.622689:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830091.622694:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830091.622707:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280968704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830091.622714:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830091.622718:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830091.622719:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830091.622723:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4641 00000100:00080000:3.0:1787830091.622726:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830091.622730:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830091.622734:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830091.622742:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280968832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830091.622862:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280968320:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830091.622951:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830091.623018:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830091/real 1787830091] req@ffff8ab3d0a421c0 x1874671280968320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830091.623044:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280968320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830091.623060:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830091.623069:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830091.623073:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280968320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830091.623081:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280968448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830091.623104:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830091.623117:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830091/real 1787830091] req@ffff8ab3d0a40000 x1874671280968448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830091.623129:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280968448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830091.623138:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830091.623141:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830091.623144:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280968448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830091.623150:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280968576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830091.623170:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830091.623245:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830091/real 1787830091] req@ffff8ab3d0a43c00 x1874671280968576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830091.623264:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280968576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830091.623272:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830091.623277:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830091.623280:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280968576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830091.623286:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280968704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830091.623313:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830091.623325:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830091/real 1787830091] req@ffff8ab3d0a41e00 x1874671280968704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830091.623336:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280968704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830091.623346:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830091.623349:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830091.623352:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280968704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830091.623358:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280968832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830091.623377:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830091.623388:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830091/real 1787830091] req@ffff8ab3d0a430c0 x1874671280968832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830091.623399:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280968832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830091.623408:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830091.623411:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830091.623414:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280968832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830092.646372:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830093.670832:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830094.694119:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830095.720317:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830096.742051:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830096.742062:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830096.742068:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830096.742080:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4646 00000100:00080000:0.0:1787830096.742092:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830096.742101:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830096.742149:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280968960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830096.742177:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830096.742181:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830096.742183:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830096.742187:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4646 00000100:00080000:0.0:1787830096.742191:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830096.742196:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830096.742207:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830096.742219:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280969088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830096.742229:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830096.742232:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830096.742234:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830096.742238:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4646 00000100:00080000:0.0:1787830096.742241:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830096.742245:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830096.742250:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830096.742261:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280969216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830096.742270:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830096.742274:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830096.742275:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830096.742279:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4646 00000100:00080000:0.0:1787830096.742282:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830096.742285:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830096.742290:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830096.742298:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280969344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830096.742307:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830096.742311:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830096.742312:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830096.742316:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4646 00000100:00080000:0.0:1787830096.742319:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830096.742323:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830096.742327:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830096.742335:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280969472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830096.742682:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280968960:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830096.742765:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830096.742828:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830096/real 1787830096] req@ffff8ab3f075f0c0 x1874671280968960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830096.742850:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280968960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830096.742871:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830096.742877:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830096.742881:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280968960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830096.742911:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280969088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830096.742938:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830096.742952:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830096/real 1787830096] req@ffff8ab3f075f480 x1874671280969088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830096.742965:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280969088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830096.742973:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830096.742976:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830096.742979:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280969088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830096.742985:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280969216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830096.743016:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830096.743027:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830096/real 1787830096] req@ffff8ab3f075ed00 x1874671280969216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830096.743039:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280969216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830096.743053:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830096.743056:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830096.743059:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280969216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830096.743065:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280969344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830096.743085:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830096.743096:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830096/real 1787830096] req@ffff8ab3f075da40 x1874671280969344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830096.743107:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280969344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830096.743118:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830096.743122:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830096.743125:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280969344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830096.743131:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280969472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830096.743151:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830096.743161:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830096/real 1787830096] req@ffff8ab3f075f840 x1874671280969472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830096.743172:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280969472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830096.743183:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830096.743186:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830096.743189:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280969472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830096.743320:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830097.766115:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830098.790181:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830099.814122:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830100.838243:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830101.862017:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830101.862032:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830101.862038:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830101.862050:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4651 00000100:00080000:0.0:1787830101.862061:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830101.862070:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830101.862119:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280969600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787830101.862137:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830101.862150:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830101.862154:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830101.862156:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830101.862160:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4651 00000100:00080000:0.0:1787830101.862164:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830101.862169:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830101.862181:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830101.862196:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280969600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830101.862198:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280969728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830101.862210:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830101.862214:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830101.862215:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830101.862219:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4651 00000100:00080000:0.0:1787830101.862222:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830101.862227:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830101.862231:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830101.862244:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280969856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830101.862253:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830101.862256:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830101.862258:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830101.862262:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4651 00000100:00080000:0.0:1787830101.862272:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830101.862277:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830101.862282:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830101.862293:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280969984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830101.862297:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830101.862308:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830101.862312:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830101.862314:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830101.862319:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4651 00000100:00080000:0.0:1787830101.862327:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830101.862332:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830101.862337:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830101.862350:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280970112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830101.862363:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830101/real 1787830101] req@ffff8ab3f075e580 x1874671280969600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830101.862385:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280969600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830101.862397:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830101.862401:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830101.862404:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280969600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830101.862452:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280969728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830101.862472:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830101.862622:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830101/real 1787830101] req@ffff8ab3f075f840 x1874671280969728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830101.862636:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280969728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830101.862645:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830101.862716:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830101.862721:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280969728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830101.862727:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280969856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830101.862749:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830101.862762:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830101/real 1787830101] req@ffff8ab3f075da40 x1874671280969856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830101.862772:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280969856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830101.862779:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830101.862783:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830101.862785:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280969856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830101.862791:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280969984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830101.862809:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830101.862827:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830101/real 1787830101] req@ffff8ab3f075ed00 x1874671280969984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830101.862842:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280969984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830101.862852:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830101.862855:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830101.862858:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280969984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830101.862864:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280970112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830101.862882:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830101.862919:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830101/real 1787830101] req@ffff8ab3f075f480 x1874671280970112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830101.862939:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280970112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830101.862950:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830101.862967:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830101.862970:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280970112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830102.886219:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830103.910373:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830104.935503:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830105.959546:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830106.984272:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830106.988087:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830106.988100:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830106.988104:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830106.988116:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4656 00000100:00080000:3.0:1787830106.988127:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830106.988135:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830106.988182:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280970240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830106.988222:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830106.988225:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830106.988227:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830106.988231:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4656 00000100:00080000:3.0:1787830106.988234:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830106.988238:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830106.988249:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830106.988261:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280970368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830106.988270:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830106.988274:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830106.988276:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830106.988280:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4656 00000100:00080000:3.0:1787830106.988283:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830106.988287:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830106.988292:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830106.988303:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280970496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830106.988312:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830106.988316:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830106.988318:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830106.988322:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4656 00000100:00080000:3.0:1787830106.988325:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830106.988330:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830106.988335:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830106.988347:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280970624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830106.988356:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830106.988359:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830106.988361:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830106.988365:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4656 00000100:00080000:3.0:1787830106.988369:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830106.988373:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830106.988377:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830106.988387:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280970752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830106.988491:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280970240:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830106.988577:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830106.988634:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830106/real 1787830106] req@ffff8ab3d0a412c0 x1874671280970240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830106.988653:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280970240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830106.988663:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830106.988668:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830106.988672:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280970240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830106.988679:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280970368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830106.988702:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830106.988715:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830106/real 1787830106] req@ffff8ab3d0a43840 x1874671280970368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830106.988728:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280970368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830106.988736:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830106.988740:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830106.988743:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280970368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830106.988749:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280970496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830106.988765:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830106.988776:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830106/real 1787830106] req@ffff8ab3d0a43480 x1874671280970496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830106.988786:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280970496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830106.988793:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830106.988850:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830106.988854:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280970496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830106.988860:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280970624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830106.988879:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830106.988932:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830106/real 1787830106] req@ffff8ab3d0a430c0 x1874671280970624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830106.988945:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280970624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830106.988954:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830106.988957:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830106.988960:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280970624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830106.988966:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280970752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830106.988984:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830106.988996:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830106/real 1787830106] req@ffff8ab3d0a41e00 x1874671280970752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830106.989008:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280970752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830106.989018:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830106.989021:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830106.989024:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280970752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830108.007027:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830109.030097:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830110.057612:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830111.079082:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830112.102103:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830112.102117:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830112.102123:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830112.102136:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4661 00000100:00080000:0.0:1787830112.102148:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830112.102157:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830112.102207:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280970880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830112.102232:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830112.102237:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830112.102239:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830112.102244:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4661 00000100:00080000:0.0:1787830112.102248:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830112.102253:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830112.102265:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830112.102265:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280970880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830112.102283:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280971008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830112.102296:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830112.102301:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830112.102303:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830112.102308:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4661 00000100:00080000:0.0:1787830112.102312:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830112.102317:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830112.102322:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830112.102334:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280971136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830112.102356:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830112.102405:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830112/real 1787830112] req@ffff8ab3f075f0c0 x1874671280970880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830112.102428:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280970880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830112.102439:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830112.102443:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830112.102447:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280970880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830112.102470:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280971008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830112.102492:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830112.102505:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830112/real 1787830112] req@ffff8ab3f075cb40 x1874671280971008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830112.102516:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280971008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830112.102529:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830112.102532:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830112.102535:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280971008:192.168.203.124@tcp:38:kworker.0 00010000:00010000:2.0:1787830112.102648:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830112.103076:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830112.103080:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830112.103083:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830112.103091:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4661 00000100:00080000:0.0:1787830112.103098:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830112.103104:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830112.103112:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830112.103118:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280971136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830112.103135:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280971264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830112.103149:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787830112.103157:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830112.103159:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830112.103163:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830112.103171:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4661 00000100:00080000:0.0:1787830112.103175:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830112.103182:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830112/real 1787830112] req@ffff8ab3f075c000 x1874671280971136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830112.103183:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830112.103194:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830112.103221:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280971136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830112.103222:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280971392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830112.103294:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830112.103301:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830112.103306:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280971136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830112.103382:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280971264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830112.103406:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830112.103418:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830112/real 1787830112] req@ffff8ab3f075de00 x1874671280971264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830112.103430:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280971264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830112.103440:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830112.103443:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830112.103446:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280971264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830112.103452:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280971392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830112.103470:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830112.103481:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830112/real 1787830112] req@ffff8ab3f075e940 x1874671280971392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830112.103492:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280971392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830112.103500:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830112.103503:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830112.103506:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280971392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830113.126148:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830114.150113:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830115.174098:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830116.198126:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830117.222104:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830117.222151:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830117.222157:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830117.222162:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830117.222171:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4666 00000100:00080000:3.0:1787830117.222182:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830117.222190:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830117.222228:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280971520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830117.222266:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830117.222270:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830117.222272:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830117.222276:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4666 00000100:00080000:3.0:1787830117.222280:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830117.222284:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830117.222293:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830117.222307:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280971648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830117.222318:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830117.222321:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830117.222323:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830117.222327:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4666 00000100:00080000:3.0:1787830117.222330:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830117.222334:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830117.222339:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830117.222350:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280971776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830117.222359:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830117.222363:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830117.222365:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830117.222369:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4666 00000100:00080000:3.0:1787830117.222373:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830117.222378:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830117.222384:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830117.222411:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280971904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830117.222420:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830117.222424:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830117.222426:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830117.222430:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4666 00000100:00080000:3.0:1787830117.222434:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830117.222438:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830117.222444:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830117.222455:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280972032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830117.222546:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280971520:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830117.222604:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830117.222643:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830117/real 1787830117] req@ffff8ab3d0a43c00 x1874671280971520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830117.222659:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280971520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830117.222679:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830117.222684:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830117.222688:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280971520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830117.222695:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280971648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830117.222721:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830117.222735:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830117/real 1787830117] req@ffff8ab3d0a40000 x1874671280971648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830117.222746:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280971648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830117.222753:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830117.222756:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830117.222759:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280971648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830117.222764:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280971776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830117.222782:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830117.222792:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830117/real 1787830117] req@ffff8ab3d0a421c0 x1874671280971776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830117.222802:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280971776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830117.222810:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830117.222813:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830117.222816:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280971776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830117.222821:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280971904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830117.222839:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830117.222864:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830117/real 1787830117] req@ffff8ab3d0a41e00 x1874671280971904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830117.222875:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280971904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830117.222883:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830117.222907:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830117.222910:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280971904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830117.222916:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280972032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830117.222934:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830117.222945:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830117/real 1787830117] req@ffff8ab3d0a430c0 x1874671280972032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830117.222955:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280972032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830117.222964:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830117.222967:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830117.222970:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280972032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830118.246299:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830119.270106:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830120.296102:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830121.318678:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830122.342104:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830122.342252:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830122.342263:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830122.342269:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830122.342282:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4671 00000100:00080000:2.0:1787830122.342300:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830122.342309:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830122.342370:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280972160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830122.343951:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830122.343956:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830122.343958:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830122.343965:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4671 00000100:00080000:2.0:1787830122.343972:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830122.343978:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830122.343992:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830122.344015:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280972288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830122.344024:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830122.344027:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830122.344029:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830122.344033:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4671 00000100:00080000:2.0:1787830122.344036:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830122.344040:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830122.344045:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830122.344055:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280972416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830122.344072:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830122.344076:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830122.344077:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830122.344081:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4671 00000100:00080000:2.0:1787830122.344084:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830122.344088:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830122.344093:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830122.344104:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280972544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830122.344109:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280972160:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830122.344180:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830122.344234:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830122/real 1787830122] req@ffff8ab3f0439a40 x1874671280972160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830122.344252:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280972160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830122.344262:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830122.344267:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830122.344270:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280972160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830122.344278:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280972288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830122.344300:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830122.344313:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830122/real 1787830122] req@ffff8ab3f043bc00 x1874671280972288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830122.344325:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280972288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830122.344334:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830122.344335:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830122.344337:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787830122.344339:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787830122.344340:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280972288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830122.344346:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280972416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830122.344364:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830122.344375:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830122/real 1787830122] req@ffff8ab3f04383c0 x1874671280972416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830122.344386:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280972416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830122.344394:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830122.344398:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830122.344401:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280972416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830122.344429:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280972544:192.168.203.124@tcp:8:kworker.0 00000100:00080000:2.0:1787830122.344437:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787830122.344449:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830122.344460:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830122/real 1787830122] req@ffff8ab3f043ad00 x1874671280972544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830122.344471:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280972544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830122.344481:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830122.344484:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830122.344487:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280972544:192.168.203.124@tcp:8:kworker.0 00000100:00080000:2.0:1787830122.344498:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4671 00000100:00080000:2.0:1787830122.344502:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830122.344507:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830122.344513:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830122.344525:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280972672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830122.344565:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280972672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830122.344584:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830122.344593:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830122/real 1787830122] req@ffff8ab3f04392c0 x1874671280972672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830122.344600:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280972672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830122.344608:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830122.344610:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830122.344611:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280972672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830123.366113:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830124.390146:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830125.414270:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830126.438194:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830127.462530:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830127.464643:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830127.464654:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830127.464659:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830127.464672:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4677 00000100:00080000:3.0:1787830127.464684:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830127.464694:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830127.464746:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280972800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830127.464786:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830127.464790:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830127.464792:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830127.464795:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4677 00000100:00080000:3.0:1787830127.464799:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830127.464803:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830127.464814:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830127.464826:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280972928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830127.464835:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830127.464838:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830127.464839:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830127.464843:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4677 00000100:00080000:3.0:1787830127.464846:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830127.464850:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830127.464854:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830127.464864:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280973056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830127.464873:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830127.464876:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830127.464878:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830127.464882:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4677 00000100:00080000:3.0:1787830127.464919:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830127.464924:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830127.464928:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830127.464941:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280973184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830127.464950:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830127.464953:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830127.464955:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830127.464959:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4677 00000100:00080000:3.0:1787830127.464961:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830127.464965:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830127.464969:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830127.464977:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280973312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830127.465077:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280972800:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830127.465155:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830127.465219:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830127/real 1787830127] req@ffff8ab3d0a43480 x1874671280972800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830127.465246:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280972800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830127.465257:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830127.465268:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830127.465271:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280972800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830127.465279:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280972928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830127.465304:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830127.465318:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830127/real 1787830127] req@ffff8ab3d0a43840 x1874671280972928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830127.465330:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280972928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830127.465338:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830127.465341:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830127.465344:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280972928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830127.465350:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280973056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830127.465367:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830127.465378:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830127/real 1787830127] req@ffff8ab3d0a412c0 x1874671280973056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830127.465388:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280973056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830127.465396:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830127.465399:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830127.465402:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280973056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830127.465407:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280973184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830127.465438:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830127.465461:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830127/real 1787830127] req@ffff8ab3d0a430c0 x1874671280973184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830127.465472:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280973184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830127.465486:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830127.465490:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830127.465500:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280973184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830127.465507:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280973312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830127.465525:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830127.465535:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830127/real 1787830127] req@ffff8ab3d0a41e00 x1874671280973312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830127.465546:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280973312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830127.465555:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830127.465557:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830127.465560:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280973312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830128.486144:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830129.510375:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830130.534810:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830131.558102:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830132.582038:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830132.582058:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830132.582064:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830132.582076:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4682 00000100:00080000:3.0:1787830132.582090:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830132.582108:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830132.582158:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280973440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830132.582206:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830132.582210:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830132.582211:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830132.582215:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4682 00000100:00080000:3.0:1787830132.582218:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830132.582230:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830132.582243:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830132.582256:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280973440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830132.582264:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280973568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830132.582280:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830132.582284:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830132.582288:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830132.582294:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4682 00010000:00010000:0.0:1787830132.582299:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830132.582300:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830132.582309:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830132.582316:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830132.582329:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280973696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830132.582341:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830132.582343:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830132.582348:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830132.582351:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830132.582358:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4682 00000100:00080000:3.0:1787830132.582363:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830132.582367:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830132.582373:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830132.582390:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280973824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830132.582400:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830132.582404:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830132/real 1787830132] req@ffff8ab3d0a421c0 x1874671280973440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830132.582406:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830132.582410:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830132.582424:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4682 00000100:00080000:3.0:1787830132.582431:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830132.582438:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830132.582446:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830132.582460:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280973952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830132.582462:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280973440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830132.582507:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830132.582514:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830132.582519:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280973440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830132.582544:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280973568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830132.582567:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830132.582581:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830132/real 1787830132] req@ffff8ab3d0a40000 x1874671280973568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830132.582595:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280973568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830132.582602:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830132.582606:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830132.582609:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280973568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830132.582616:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280973696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830132.582636:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830132.582647:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830132/real 1787830132] req@ffff8ab3d0a43c00 x1874671280973696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830132.582658:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280973696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830132.582665:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830132.582669:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830132.582672:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280973696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830132.582679:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280973824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830132.582699:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830132.582710:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830132/real 1787830132] req@ffff8ab3d0a41e00 x1874671280973824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830132.582721:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280973824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830132.582730:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830132.582733:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830132.582736:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280973824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830132.582755:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280973952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830132.582773:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830132.582784:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830132/real 1787830132] req@ffff8ab3d0a430c0 x1874671280973952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830132.582795:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280973952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830132.582804:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830132.582808:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830132.582811:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280973952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830133.606122:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830134.632815:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830135.654138:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830136.678659:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830137.702003:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830137.702019:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830137.702024:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830137.702034:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4687 00000100:00080000:3.0:1787830137.702043:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830137.702050:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830137.702087:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280974080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830137.702112:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830137.702116:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830137.702118:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830137.702121:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4687 00000100:00080000:3.0:1787830137.702125:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830137.702129:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830137.702138:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830137.702150:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280974208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830137.702159:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830137.702162:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830137.702164:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830137.702168:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4687 00000100:00080000:3.0:1787830137.702171:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830137.702180:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830137.702187:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830137.702207:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280974336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830137.702216:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830137.702220:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830137.702222:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830137.702227:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4687 00000100:00080000:3.0:1787830137.702230:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830137.702240:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830137.702246:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830137.702257:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280974464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830137.702267:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830137.702270:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830137.702272:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830137.702276:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4687 00000100:00080000:3.0:1787830137.702279:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830137.702282:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280974080:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787830137.702285:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830137.702292:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830137.702302:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280974592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830137.702335:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830137.702387:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830137/real 1787830137] req@ffff8ab3d0a412c0 x1874671280974080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830137.702404:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280974080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830137.702414:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830137.702419:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830137.702423:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280974080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830137.702429:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280974208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830137.702445:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830137.702454:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830137/real 1787830137] req@ffff8ab3d0a43840 x1874671280974208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830137.702461:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280974208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830137.702466:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830137.702468:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830137.702470:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280974208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830137.702473:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280974336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830137.702482:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830137.702486:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830137/real 1787830137] req@ffff8ab3d0a43480 x1874671280974336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830137.702493:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280974336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830137.702501:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830137.702503:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830137.702505:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280974336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830137.702509:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280974464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830137.702518:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830137.702523:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830137/real 1787830137] req@ffff8ab3d0a430c0 x1874671280974464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830137.702529:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280974464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830137.702535:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830137.702537:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830137.702538:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280974464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830137.702567:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280974592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830137.702578:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830137.702584:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830137/real 1787830137] req@ffff8ab3d0a41e00 x1874671280974592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830137.702590:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280974592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830137.702595:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830137.702597:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830137.702599:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280974592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830137.702654:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830138.726834:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830139.750103:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830140.774569:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830141.798143:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830142.822036:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830142.822046:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830142.822050:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830142.822062:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4692 00000100:00080000:3.0:1787830142.822072:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830142.822080:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830142.822120:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280974720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830142.822147:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830142.822150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830142.822152:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830142.822155:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4692 00000100:00080000:3.0:1787830142.822160:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830142.822164:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830142.822173:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830142.822185:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280974848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830142.822194:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830142.822197:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830142.822199:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830142.822203:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4692 00000100:00080000:3.0:1787830142.822206:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830142.822210:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830142.822215:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830142.822225:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280974976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830142.822234:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830142.822237:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830142.822238:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830142.822242:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4692 00000100:00080000:3.0:1787830142.822245:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830142.822249:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830142.822254:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830142.822266:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280975104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830142.822280:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830142.822283:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830142.822284:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830142.822288:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4692 00000100:00080000:3.0:1787830142.822292:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830142.822295:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830142.822299:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830142.822309:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280975232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830142.822478:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280974720:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830142.822539:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830142.822597:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830142/real 1787830142] req@ffff8ab3d0a43c00 x1874671280974720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830142.822614:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280974720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830142.822634:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830142.822638:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830142.822644:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280974720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830142.822653:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280974848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830142.822687:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830142.822701:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830142/real 1787830142] req@ffff8ab3d0a40000 x1874671280974848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830142.822713:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280974848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1787830142.822718:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830142.822722:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830142.822727:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830142.822732:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280974848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830142.822738:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280974976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830142.822756:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830142.822767:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830142/real 1787830142] req@ffff8ab3d0a421c0 x1874671280974976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830142.822777:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280974976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830142.822784:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830142.822787:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830142.822790:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280974976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830142.822795:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280975104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830142.822813:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830142.822828:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830142/real 1787830142] req@ffff8ab3d0a41e00 x1874671280975104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830142.822839:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280975104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830142.822848:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830142.822850:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830142.822853:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280975104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830142.822858:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280975232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830142.822872:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830142.822880:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830142/real 1787830142] req@ffff8ab3d0a430c0 x1874671280975232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830142.822909:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280975232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830142.822917:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830142.822920:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830142.822923:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280975232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830143.847928:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830144.870187:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830145.894241:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830146.918961:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830147.943746:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830147.943756:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830147.943760:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830147.943771:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4697 00000100:00080000:1.0:1787830147.943781:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830147.943789:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830147.943823:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280975360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830147.943859:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830147.943863:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830147.943864:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830147.943868:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4697 00000100:00080000:1.0:1787830147.943872:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830147.943876:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830147.943916:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830147.943929:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280975488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830147.943939:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830147.943942:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830147.943944:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830147.943948:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4697 00000100:00080000:1.0:1787830147.943951:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830147.943956:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830147.943960:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830147.943971:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280975616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830147.943979:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830147.943982:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830147.943983:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830147.943987:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4697 00000100:00080000:1.0:1787830147.943989:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830147.943993:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830147.943998:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830147.944006:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280975744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830147.944014:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830147.944017:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830147.944019:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830147.944022:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4697 00000100:00080000:1.0:1787830147.944025:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830147.944029:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830147.944032:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830147.944042:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280975872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787830147.944579:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830147.945321:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280975360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830147.945434:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280975488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830147.945467:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280975616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830147.945491:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280975744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830147.945514:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280975872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830148.966365:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830149.990084:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830151.014114:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787830151.014164:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787830151.014203:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787830151.014220:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787830151.014257:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787830151.014267:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787830151.014274:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787830151.014282:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:3.0:1787830151.014444:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830147/real 1787830151] req@ffff8ab3d22fd680 x1874671280975360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830151.014467:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280975360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830151.014478:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830151.014483:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830151.014487:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280975360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830151.014495:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830147/real 1787830151] req@ffff8ab3d22fcb40 x1874671280975488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830151.014507:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280975488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830151.014515:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830151.014519:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830151.014522:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280975488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830151.014529:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830147/real 1787830151] req@ffff8ab3d22fda40 x1874671280975616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830151.014540:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280975616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830151.014547:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830151.014550:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830151.014553:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280975616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830151.014560:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830147/real 1787830151] req@ffff8ab3d22fc000 x1874671280975744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830151.014571:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280975744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830151.014580:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830151.014584:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830151.014586:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280975744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830151.014592:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830147/real 1787830151] req@ffff8ab3d22ff480 x1874671280975872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830151.014604:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280975872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830151.014626:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830151.014629:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830151.014633:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280975872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830152.039397:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830153.062082:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830153.063306:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830153.063317:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830153.063322:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830153.063334:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4702 00000100:00080000:3.0:1787830153.063346:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830153.063354:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830153.063397:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280976000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830153.063433:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830153.063437:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830153.063439:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830153.063442:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4702 00000100:00080000:3.0:1787830153.063446:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830153.063450:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830153.063462:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830153.063473:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280976128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830153.063481:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830153.063484:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830153.063486:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830153.063489:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4702 00000100:00080000:3.0:1787830153.063492:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830153.063495:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830153.063499:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830153.063507:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280976256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830153.063514:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830153.063517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830153.063519:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830153.063523:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4702 00000100:00080000:3.0:1787830153.063525:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830153.063529:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830153.063534:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830153.063545:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280976384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830153.063553:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830153.063557:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830153.063559:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830153.063563:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4702 00000100:00080000:3.0:1787830153.063566:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830153.063570:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830153.063574:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830153.063589:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280976512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830153.063879:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280976000:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830153.063967:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830153.064031:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830153/real 1787830153] req@ffff8ab3d0a43480 x1874671280976000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830153.064050:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280976000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830153.064061:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830153.064065:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830153.064069:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280976000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830153.064077:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280976128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830153.064100:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830153.064113:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830153/real 1787830153] req@ffff8ab3d0a43840 x1874671280976128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830153.064125:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280976128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830153.064133:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830153.064136:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830153.064139:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280976128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830153.064145:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280976256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830153.064164:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830153.064174:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830153/real 1787830153] req@ffff8ab3d0a412c0 x1874671280976256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830153.064184:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280976256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830153.064191:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830153.064194:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830153.064196:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280976256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830153.064201:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280976384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830153.064218:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830153.064227:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830153/real 1787830153] req@ffff8ab3d0a430c0 x1874671280976384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830153.064237:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280976384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830153.064247:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830153.064250:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830153.064253:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280976384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830153.064258:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280976512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830153.064273:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830153.064283:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830153/real 1787830153] req@ffff8ab3d0a41e00 x1874671280976512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830153.064300:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280976512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830153.064308:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830153.064311:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830153.064314:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280976512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830154.086153:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830155.112010:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830156.134436:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830157.158189:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830158.182039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830158.182052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830158.182057:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830158.182071:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4707 00000100:00080000:0.0:1787830158.182083:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830158.182092:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830158.182151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280976640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830158.182176:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830158.182179:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830158.182181:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830158.182332:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4707 00000100:00080000:0.0:1787830158.182338:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830158.182343:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830158.182354:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830158.182371:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280976768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830158.182380:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830158.182383:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830158.182385:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830158.182389:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4707 00000100:00080000:0.0:1787830158.182392:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830158.182396:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830158.182404:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830158.182414:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280976896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830158.182423:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830158.182426:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830158.182428:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830158.182432:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4707 00000100:00080000:0.0:1787830158.182435:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830158.182439:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830158.182444:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830158.182453:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280977024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830158.182462:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830158.182465:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830158.182467:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830158.182473:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4707 00000100:00100000:1.0:1787830158.182475:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280976640:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830158.182476:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830158.182483:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830158.182487:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830158.182496:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280977152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830158.182550:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830158.182683:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830158/real 1787830158] req@ffff8ab3f075cf00 x1874671280976640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830158.182702:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280976640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830158.182713:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830158.182717:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830158.182721:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280976640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830158.182727:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280976768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830158.182748:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830158.182760:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830158/real 1787830158] req@ffff8ab3f075e940 x1874671280976768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830158.182770:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280976768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830158.182791:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830158.182793:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830158.182796:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280976768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830158.182801:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280976896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830158.182817:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830158.182827:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830158/real 1787830158] req@ffff8ab3f075de00 x1874671280976896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830158.182836:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280976896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830158.182842:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830158.182845:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830158.182848:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280976896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830158.182874:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280977024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830158.182918:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830158.182929:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830158/real 1787830158] req@ffff8ab3f075c000 x1874671280977024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830158.182941:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280977024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830158.182951:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830158.182954:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830158.182957:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280977024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830158.182979:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280977152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830158.182996:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830158.183005:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830158/real 1787830158] req@ffff8ab3f075cb40 x1874671280977152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830158.183015:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280977152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830158.183024:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830158.183027:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830158.183030:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280977152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830158.183190:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830159.206482:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830160.230141:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830161.254702:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830162.278174:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830163.303430:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830163.303440:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830163.303444:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830163.303455:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4712 00000100:00080000:1.0:1787830163.303465:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830163.303472:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830163.303516:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280977280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830163.303747:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830163.303751:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830163.303753:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830163.303757:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4712 00000100:00080000:1.0:1787830163.303761:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830163.303766:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830163.303776:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830163.303791:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280977408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830163.303803:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830163.303806:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830163.303809:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830163.303813:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280977280:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787830163.303815:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4712 00000100:00080000:1.0:1787830163.303822:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830163.303827:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830163.303831:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830163.303843:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280977536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830163.303853:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830163.303856:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830163.303858:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830163.303862:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4712 00000800:00000100:3.0:1787830163.303969:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:1.0:1787830163.305770:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830163.305787:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830163.305798:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830163.305825:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280977664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830163.305839:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830163.305844:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830163.305847:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830163.305851:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4712 00000100:00080000:1.0:1787830163.305856:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830163.305859:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830163.305864:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830163.305872:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280977792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787830163.306217:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830163.306453:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830163/real 1787830163] req@ffff8ab3d22ff480 x1874671280977280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830163.306477:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280977280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830163.306487:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830163.306491:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830163.306495:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280977280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830163.306537:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280977408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830163.306588:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830163.306609:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830163/real 1787830163] req@ffff8ab3d22fc000 x1874671280977408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830163.306622:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280977408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830163.306635:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830163.306639:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830163.306642:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280977408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830163.306649:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280977536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830163.306670:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830163.306681:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830163/real 1787830163] req@ffff8ab3d22fda40 x1874671280977536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830163.306693:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280977536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830163.306701:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830163.306704:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830163.306708:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280977536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830163.306714:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280977664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830163.306733:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830163.306744:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830163/real 1787830163] req@ffff8ab3d22fcb40 x1874671280977664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830163.306756:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280977664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830163.306765:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830163.306768:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830163.306772:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280977664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830163.306777:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280977792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830163.306799:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830163.306810:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830163/real 1787830163] req@ffff8ab3d22fd680 x1874671280977792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830163.306820:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280977792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830163.306830:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830163.306833:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830163.306836:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280977792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830164.326424:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830165.350187:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830166.374152:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830167.399069:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830168.422479:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830168.422498:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830168.422503:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1787830168.422508:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830168.422514:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4718 00000100:00080000:2.0:1787830168.422525:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830168.422532:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830168.422579:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280977920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830168.422608:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830168.422611:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830168.422613:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830168.422617:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4718 00000100:00080000:2.0:1787830168.422621:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830168.422625:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830168.422634:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830168.422648:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280978048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830168.422658:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830168.422662:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830168.422664:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830168.422668:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4718 00000100:00080000:2.0:1787830168.422672:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830168.422677:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830168.422683:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830168.422694:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280977920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830168.422696:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280978176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830168.422724:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830168.422728:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830168.422730:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830168.422736:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4718 00000100:00080000:2.0:1787830168.422740:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830168.422745:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830168.422751:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830168.422765:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280978304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830168.422775:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830168.422781:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830168.422786:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830168.422788:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830168.422796:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4718 00000100:00080000:2.0:1787830168.422810:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830168.422816:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830168.422822:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830168.422839:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280978432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830168.422852:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830168/real 1787830168] req@ffff8ab3f0439680 x1874671280977920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830168.422873:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280977920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830168.422934:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830168.422939:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830168.422944:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280977920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830168.422952:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280978048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830168.422979:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830168.422992:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830168/real 1787830168] req@ffff8ab3f043b0c0 x1874671280978048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830168.423005:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280978048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830168.423012:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830168.423016:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830168.423019:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280978048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830168.423051:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280978176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830168.423075:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830168.423084:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830168/real 1787830168] req@ffff8ab3f043a580 x1874671280978176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830168.423091:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280978176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830168.423097:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830168.423099:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830168.423101:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280978176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830168.423105:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280978304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830168.423124:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830168.423134:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830168/real 1787830168] req@ffff8ab3f0438f00 x1874671280978304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830168.423151:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280978304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830168.423174:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830168.423177:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830168.423187:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280978304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830168.423193:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280978432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830168.423214:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830168.423225:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830168/real 1787830168] req@ffff8ab3f0438b40 x1874671280978432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830168.423237:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280978432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830168.423246:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830168.423249:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830168.423252:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280978432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830169.446057:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830170.471182:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830171.495946:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830172.518243:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830173.542151:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830173.542166:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830173.542172:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830173.542185:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4723 00000100:00080000:2.0:1787830173.542198:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830173.542206:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830173.542269:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671280978560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830173.542304:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830173.542308:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830173.542310:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830173.542314:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4723 00000100:00080000:2.0:1787830173.542317:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830173.542321:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830173.542333:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830173.542346:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280978688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830173.542355:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830173.542358:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830173.542361:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830173.542365:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4723 00000100:00080000:2.0:1787830173.542368:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830173.542371:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830173.542376:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830173.542386:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280978816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830173.542394:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830173.542398:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830173.542399:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830173.542403:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4723 00000100:00080000:2.0:1787830173.542406:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830173.542411:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830173.542416:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830173.542427:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280978944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830173.542438:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830173.542442:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830173.542444:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830173.542449:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4723 00000100:00100000:3.0:1787830173.542465:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280978560:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787830173.542467:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830173.542475:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830173.542481:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830173.542494:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280979072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830173.542550:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830173.542610:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830173/real 1787830173] req@ffff8ab3f0438780 x1874671280978560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830173.542631:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671280978560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830173.542651:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830173.542656:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830173.542661:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280978560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830173.542668:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280978688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830173.542709:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830173.542725:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830173/real 1787830173] req@ffff8ab3f0438b40 x1874671280978688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830173.542738:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280978688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830173.542747:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830173.542751:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830173.542754:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280978688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830173.542761:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280978816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830173.542780:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830173.542792:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830173/real 1787830173] req@ffff8ab3f0438f00 x1874671280978816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830173.542803:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280978816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830173.542811:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830173.542815:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830173.542819:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280978816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830173.542825:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280978944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830173.542844:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830173.542853:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830173/real 1787830173] req@ffff8ab3f043a580 x1874671280978944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830173.542865:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280978944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830173.542875:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830173.542878:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830173.542882:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280978944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830173.542943:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280979072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830173.542962:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830173.542973:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830173/real 1787830173] req@ffff8ab3f043b0c0 x1874671280979072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830173.542984:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280979072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830173.542993:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830173.542997:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830173.543000:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280979072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830173.543636:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830174.566200:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830175.590508:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830176.614483:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830177.638191:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830178.662023:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830178.662045:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830178.662051:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830178.662062:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4728 00000100:00080000:2.0:1787830178.662072:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830178.662080:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830178.662120:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280979200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830178.662151:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830178.662155:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830178.662157:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830178.662161:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4728 00000100:00080000:2.0:1787830178.662165:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830178.662170:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830178.662179:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830178.662193:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280979328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830178.662203:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830178.662206:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830178.662212:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830178.662216:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4728 00000100:00080000:2.0:1787830178.662220:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1787830178.662225:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830178.662225:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830178.662231:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830178.662255:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280979456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830178.662264:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830178.662267:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830178.662269:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830178.662272:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4728 00000100:00080000:2.0:1787830178.662275:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830178.662280:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830178.662284:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830178.662295:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280979584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830178.662303:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830178.662307:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830178.662309:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830178.662314:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4728 00000100:00080000:2.0:1787830178.662317:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830178.662321:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830178.662326:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830178.662331:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280979200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830178.662339:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280979712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830178.662407:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830178.662498:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830178/real 1787830178] req@ffff8ab3f0439680 x1874671280979200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830178.662526:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280979200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830178.662537:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830178.662542:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830178.662546:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280979200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830178.662553:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280979328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830178.662576:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830178.662595:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830178/real 1787830178] req@ffff8ab3f04392c0 x1874671280979328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830178.662608:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280979328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830178.662616:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830178.662619:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830178.662622:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280979328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830178.662628:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280979456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830178.662648:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830178.662660:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830178/real 1787830178] req@ffff8ab3f043ad00 x1874671280979456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830178.662673:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280979456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830178.662686:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830178.662689:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830178.662692:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280979456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830178.662698:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280979584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830178.662716:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830178.662726:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830178/real 1787830178] req@ffff8ab3f04383c0 x1874671280979584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830178.662737:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280979584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830178.662746:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830178.662749:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830178.662752:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280979584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830178.662778:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280979712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830178.662797:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830178.662808:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830178/real 1787830178] req@ffff8ab3f043bc00 x1874671280979712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830178.662819:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280979712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830178.662828:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830178.662831:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830178.662835:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280979712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830179.686170:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830180.710420:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830181.734462:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830182.758149:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830183.782461:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830183.782472:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830183.782477:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830183.782489:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4733 00000100:00080000:0.0:1787830183.782499:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830183.782506:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830183.782543:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280979840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787830183.782543:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830183.782578:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830183.782582:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830183.782584:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830183.782587:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4733 00000100:00080000:0.0:1787830183.782591:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830183.782599:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830183.782610:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830183.782630:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280979840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830183.782631:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280979968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830183.782659:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830183.782662:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830183.782664:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830183.782669:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4733 00000100:00080000:0.0:1787830183.782673:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830183.782677:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830183.782682:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830183.782695:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280980096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830183.782704:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830183.782706:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830183.782708:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830183.782711:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4733 00000800:00000100:3.0:1787830183.783846:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830183.785473:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830183.785491:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830183.785502:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830183.785529:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280980224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830183.785544:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830183.785548:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830183.785550:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830183.785554:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4733 00000100:00080000:0.0:1787830183.785559:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830183.785563:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830183.785566:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830183.785575:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280980352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830183.786482:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830183/real 1787830183] req@ffff8ab3f075f0c0 x1874671280979840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830183.786511:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280979840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830183.786524:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830183.786529:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830183.786533:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280979840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830183.786571:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280979968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830183.786630:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830183.786645:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830183/real 1787830183] req@ffff8ab3f075f480 x1874671280979968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830183.786658:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280979968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830183.786667:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830183.786670:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830183.786674:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280979968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830183.786681:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280980096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830183.786702:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830183.786712:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830183/real 1787830183] req@ffff8ab3f075ed00 x1874671280980096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830183.787410:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280980096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830183.787420:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830183.787425:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830183.787428:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280980096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830183.787437:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280980224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830183.787466:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830183.787481:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830183/real 1787830183] req@ffff8ab3f075da40 x1874671280980224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830183.787493:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280980224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830183.787504:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830183.787508:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830183.787511:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280980224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830183.787518:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280980352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830183.787535:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830183.787545:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830183/real 1787830183] req@ffff8ab3f075f840 x1874671280980352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830183.787563:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280980352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830183.787573:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830183.787576:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830183.787579:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280980352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830184.807269:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830185.830402:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830186.854245:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830187.878151:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830188.901990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830188.901998:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830188.902002:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830188.902012:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4738 00000100:00080000:0.0:1787830188.902022:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830188.902029:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830188.902066:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280980480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830188.902140:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280980480:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830188.902188:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830188.902215:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830188.902218:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830188.902219:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830188.902222:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4738 00000100:00080000:0.0:1787830188.902225:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830188.902228:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830188.902235:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830188.902246:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280980608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830188.902371:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830188/real 1787830188] req@ffff8ab3f075e580 x1874671280980480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830188.902384:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280980480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830188.902390:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830188.902393:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830188.902395:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280980480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830188.902412:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280980608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830188.902429:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830188.902436:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830188/real 1787830188] req@ffff8ab3f075f840 x1874671280980608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830188.902443:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280980608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830188.902449:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830188.902460:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830188.902462:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280980608:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787830188.903056:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830188.903062:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830188.903065:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830188.903071:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4738 00000100:00080000:0.0:1787830188.903078:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830188.903083:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830188.903089:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00010000:00010000:2.0:1787830188.903104:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830188.903106:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280980736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830188.903159:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280980736:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787830188.903160:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830188.903163:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830188.903165:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830188.903169:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4738 00000100:00080000:0.0:1787830188.903173:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830188.903177:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830188.903182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:3.0:1787830188.903186:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830188.903194:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280980864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830188.903199:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830188/real 1787830188] req@ffff8ab3f075da40 x1874671280980736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830188.903203:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830188.903206:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830188.903208:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830188.903210:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280980736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830188.903211:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4738 00000100:00080000:0.0:1787830188.903214:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830188.903218:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:3.0:1787830188.903218:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830188.903221:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000008:00100000:0.0:1787830188.903222:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830188.903224:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280980736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830188.903229:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280980992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830188.903236:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280980864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830188.903253:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830188.903263:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830188/real 1787830188] req@ffff8ab3f075ed00 x1874671280980864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830188.903273:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280980864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830188.903282:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830188.903285:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830188.903289:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280980864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830188.903299:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280980992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830188.903316:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830188.903326:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830188/real 1787830188] req@ffff8ab3f075f480 x1874671280980992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830188.903335:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280980992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830188.903343:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830188.903345:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830188.903359:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280980992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830189.926600:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830190.950822:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830191.974769:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830192.998988:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830194.022169:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830194.022184:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830194.022189:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830194.022202:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4743 00000100:00080000:0.0:1787830194.022216:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830194.022225:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830194.022274:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280981120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830194.022361:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280981120:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830194.022439:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830194.022533:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830194.022538:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00010000:00010000:3.0:1787830194.022539:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830194.022541:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830194.022547:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4743 00000100:00080000:0.0:1787830194.022552:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830194.022557:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830194.022569:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830194.022585:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280981248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830194.023599:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830194/real 1787830194] req@ffff8ab3f075f0c0 x1874671280981120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830194.023637:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280981120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830194.024008:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830194.024017:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830194.024023:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280981120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830194.024068:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280981248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830194.024159:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830194.024180:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830194/real 1787830194] req@ffff8ab3f075cb40 x1874671280981248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830194.024193:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280981248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830194.024202:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830194.024206:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830194.024210:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280981248:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787830194.025039:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830194.025056:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830194.025060:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830194.025067:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4743 00000100:00080000:0.0:1787830194.025075:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830194.025084:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830194.025098:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830194.025129:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280981376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830194.025144:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830194.025148:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830194.025150:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830194.025153:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4743 00000100:00080000:0.0:1787830194.025157:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830194.025162:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830194.025167:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830194.025179:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280981504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830194.025187:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830194.025191:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830194.025193:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830194.025197:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4743 00000100:00080000:0.0:1787830194.025201:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830194.025205:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830194.025209:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830194.025220:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280981632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830194.025674:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280981376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830194.025802:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830194.025830:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830194/real 1787830194] req@ffff8ab3f075c000 x1874671280981376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830194.025846:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280981376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830194.025857:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830194.025861:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830194.025865:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280981376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830194.025871:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280981504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830194.025918:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830194.025930:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830194/real 1787830194] req@ffff8ab3f075de00 x1874671280981504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830194.025942:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280981504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830194.025953:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830194.025956:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830194.025960:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280981504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830194.025967:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280981632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830194.025987:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830194.025997:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830194/real 1787830194] req@ffff8ab3f075e940 x1874671280981632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830194.026008:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280981632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830194.026017:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830194.026020:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830194.026022:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280981632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830195.046195:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830196.070137:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830197.094829:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830198.119160:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830199.142139:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830199.142335:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830199.142345:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830199.142349:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830199.142360:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4748 00000100:00080000:0.0:1787830199.142370:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830199.142377:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830199.142414:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280981760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830199.142440:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830199.142443:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830199.142445:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830199.142449:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4748 00000100:00080000:0.0:1787830199.142453:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830199.142457:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830199.142468:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830199.142494:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280981760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830199.142495:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280981888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830199.142520:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830199.142524:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830199.142526:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830199.142532:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4748 00000100:00080000:0.0:1787830199.142535:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830199.142540:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830199.142546:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830199.142558:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280982016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830199.142568:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830199.142571:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830199.142573:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830199.142576:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4748 00000100:00080000:0.0:1787830199.142583:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830199.142587:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830199.142592:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830199.142608:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280982144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830199.142619:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830199.142622:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830199.142624:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830199.142627:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4748 00000100:00080000:0.0:1787830199.142630:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830199.142634:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830199.142638:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830199.142647:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280982272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830199.143067:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830199.143129:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830199/real 1787830199] req@ffff8ab3f075cf00 x1874671280981760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830199.143149:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280981760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830199.143161:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830199.143166:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830199.143170:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280981760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830199.143197:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280981888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830199.143224:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830199.143238:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830199/real 1787830199] req@ffff8ab3f075e940 x1874671280981888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830199.143249:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280981888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830199.143257:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830199.143260:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830199.143263:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280981888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830199.143269:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280982016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830199.144410:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830199.144432:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830199/real 1787830199] req@ffff8ab3f075de00 x1874671280982016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830199.144444:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280982016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830199.144452:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830199.144456:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830199.144459:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280982016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830199.144466:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280982144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830199.144484:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830199.144493:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830199/real 1787830199] req@ffff8ab3f075c000 x1874671280982144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830199.144503:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280982144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830199.144513:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830199.144517:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830199.144519:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280982144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830199.144525:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280982272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830199.144687:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830199.144701:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830199/real 1787830199] req@ffff8ab3f075cb40 x1874671280982272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830199.144713:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280982272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830199.144724:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830199.144728:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830199.144731:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280982272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830200.167574:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830201.190113:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830202.215614:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830203.239158:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830204.262162:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1787830204.262169:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830204.262172:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830204.262180:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830204.262192:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4753 00000100:00080000:2.0:1787830204.262204:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830204.262211:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830204.262258:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280982400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830204.262290:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830204.262294:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830204.262296:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830204.262299:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4753 00000100:00080000:2.0:1787830204.262303:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830204.262307:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830204.262316:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830204.262329:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280982528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830204.262339:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830204.262342:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830204.262344:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830204.262350:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4753 00000100:00100000:1.0:1787830204.262352:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280982400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787830204.262354:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830204.262362:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830204.262367:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830204.262379:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280982656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830204.262388:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830204.262392:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830204.262393:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830204.262397:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4753 00000100:00080000:2.0:1787830204.262409:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830204.262414:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830204.262419:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787830204.262420:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830204.262434:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280982784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830204.262444:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830204.262451:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830204.262453:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830204.262457:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4753 00000100:00080000:2.0:1787830204.262460:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830204.262463:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830204.262467:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830204.262478:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280982912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830204.262526:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830204/real 1787830204] req@ffff8ab3f0439a40 x1874671280982400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830204.262545:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280982400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830204.262555:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830204.262559:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830204.262563:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280982400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830204.262570:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280982528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830204.262593:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830204.262606:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830204/real 1787830204] req@ffff8ab3f043a940 x1874671280982528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830204.262617:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280982528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830204.262625:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830204.262629:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830204.262632:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280982528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830204.262653:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280982656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830204.262671:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830204.262681:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830204/real 1787830204] req@ffff8ab3f043a1c0 x1874671280982656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830204.262692:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280982656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830204.262700:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830204.262703:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830204.262706:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280982656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830204.262711:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280982784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830204.262730:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830204.262741:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830204/real 1787830204] req@ffff8ab3f0439e00 x1874671280982784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830204.262751:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280982784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830204.262759:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830204.262762:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830204.262765:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280982784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830204.262771:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280982912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830204.262788:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830204.262799:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830204/real 1787830204] req@ffff8ab3f043b840 x1874671280982912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830204.262810:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280982912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830204.262819:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830204.262822:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830204.262825:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280982912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830205.286613:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830206.310153:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830207.334197:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830208.358150:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830209.383379:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830209.383451:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830209.383457:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830209.383461:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830209.383470:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4759 00000100:00080000:0.0:1787830209.383481:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830209.383487:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830209.383528:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280983040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830209.383557:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830209.383563:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830209.383564:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830209.383568:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4759 00000100:00080000:0.0:1787830209.383572:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830209.383577:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830209.383586:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830209.383597:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280983168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830209.383607:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830209.383609:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830209.383611:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830209.383615:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4759 00000100:00080000:0.0:1787830209.383618:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830209.383621:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830209.383626:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830209.383635:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280983296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830209.383643:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830209.383646:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830209.383647:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830209.383651:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4759 00000100:00080000:0.0:1787830209.383653:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830209.383657:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830209.383662:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830209.383671:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280983424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830209.383679:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830209.383682:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830209.383683:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830209.383687:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4759 00000100:00080000:0.0:1787830209.383689:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830209.383693:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830209.383697:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830209.383705:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280983552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830209.383785:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280983040:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830209.383917:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830209.383963:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830209/real 1787830209] req@ffff8ab3f075f0c0 x1874671280983040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830209.383979:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280983040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830209.383990:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830209.383993:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830209.383997:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280983040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830209.384004:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280983168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830209.384027:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830209.384039:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830209/real 1787830209] req@ffff8ab3f075f480 x1874671280983168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830209.384051:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280983168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830209.384059:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830209.384062:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830209.384065:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280983168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830209.384071:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280983296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830209.384089:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830209.384099:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830209/real 1787830209] req@ffff8ab3f075ed00 x1874671280983296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830209.384110:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280983296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830209.384118:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830209.384122:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830209.384125:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280983296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830209.384131:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280983424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830209.384149:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830209.384160:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830209/real 1787830209] req@ffff8ab3f075da40 x1874671280983424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830209.384171:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280983424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830209.384180:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830209.384183:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830209.384186:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280983424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830209.384192:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280983552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830209.384224:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830209.384235:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830209/real 1787830209] req@ffff8ab3f075f840 x1874671280983552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830209.384245:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280983552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830209.384254:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830209.384257:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830209.384260:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280983552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830210.406408:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830211.430166:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830212.454093:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830213.479805:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830214.502022:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830214.502041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830214.502046:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830214.502057:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4764 00010000:00010000:3.0:1787830214.502064:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830214.502068:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830214.502077:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830214.502116:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280983680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830214.502143:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830214.502147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830214.502148:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830214.502152:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4764 00000100:00080000:0.0:1787830214.502156:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830214.502161:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830214.502180:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830214.502195:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280983808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830214.502205:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830214.502208:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830214.502211:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830214.502215:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4764 00000100:00080000:0.0:1787830214.502220:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830214.502220:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280983680:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830214.502229:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830214.502235:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830214.502248:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280983936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830214.502258:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830214.502261:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830214.502262:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830214.502266:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4764 00000100:00080000:0.0:1787830214.502271:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830214.502275:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830214.502280:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830214.502291:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280984064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830214.502306:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830214.502309:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830214.502311:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830214.502316:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4764 00000100:00080000:0.0:1787830214.502320:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830214.502320:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280983808:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787830214.502329:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830214.502334:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830214.502348:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280984192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830214.502367:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280983936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830214.502391:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280984064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830214.502423:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280984192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830215.526118:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830216.551097:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830217.574724:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787830217.577316:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:1.0:1787830217.577360:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787830217.577385:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787830217.577417:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787830217.577426:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787830217.577433:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:1.0:1787830217.577441:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:3.0:1787830217.577624:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830214/real 1787830217] req@ffff8ab3f075da40 x1874671280983936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830217.577656:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280983936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830217.577668:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830217.577673:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830217.577677:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280983936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830217.577684:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830214/real 1787830217] req@ffff8ab3f075ed00 x1874671280984064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830217.577695:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280984064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830217.577705:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830217.577709:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830217.577712:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280984064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830217.577719:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830214/real 1787830217] req@ffff8ab3f075f480 x1874671280984192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830217.577730:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280984192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830217.577739:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830217.577742:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830217.577745:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280984192:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830217.577751:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830214/real 1787830217] req@ffff8ab3f075e580 x1874671280983680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830217.577763:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280983680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830217.577770:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830217.577773:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830217.577776:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280983680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830217.577782:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830214/real 1787830217] req@ffff8ab3f075f840 x1874671280983808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830217.577792:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280983808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830217.577799:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830217.577802:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830217.577805:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280983808:192.168.203.124@tcp:38:kworker.0 00010000:00010000:3.0:1787830218.600035:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830219.623047:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830219.623058:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830219.623062:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830219.623073:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4769 00000100:00080000:0.0:1787830219.623083:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830219.623091:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830219.623128:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280984320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830219.623182:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830219.623186:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830219.623188:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830219.623192:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4769 00000100:00080000:0.0:1787830219.623195:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830219.623199:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830219.623210:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830219.623222:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280984448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830219.623232:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830219.623235:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830219.623236:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830219.623240:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4769 00000100:00080000:0.0:1787830219.623243:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830219.623247:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830219.623251:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830219.623262:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280984576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830219.623271:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830219.623274:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830219.623276:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830219.623279:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4769 00000100:00080000:0.0:1787830219.623282:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830219.623286:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830219.623291:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830219.623303:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280984704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830219.623311:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830219.623314:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830219.623316:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830219.623320:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4769 00000100:00080000:0.0:1787830219.623323:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830219.623326:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830219.623331:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830219.623340:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280984832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830219.623644:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280984320:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830219.623699:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830219.623743:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830219/real 1787830219] req@ffff8ab3f075f0c0 x1874671280984320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830219.623757:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280984320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830219.623766:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830219.623769:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830219.623774:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280984320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830219.623780:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280984448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830219.623798:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830219.623807:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830219/real 1787830219] req@ffff8ab3f075cb40 x1874671280984448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830219.623817:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280984448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830219.623823:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830219.623825:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830219.623828:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280984448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830219.623832:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280984576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830219.623845:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830219.623852:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830219/real 1787830219] req@ffff8ab3f075c000 x1874671280984576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830219.623861:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280984576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830219.623867:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830219.623869:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830219.623872:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280984576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830219.623876:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280984704:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830219.625129:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1787830219.626915:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830219.626948:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830219/real 1787830219] req@ffff8ab3f075de00 x1874671280984704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830219.626967:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280984704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830219.626979:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830219.626983:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830219.626987:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280984704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830219.626997:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280984832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830219.627026:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830219.627037:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830219/real 1787830219] req@ffff8ab3f075e940 x1874671280984832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830219.627047:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280984832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830219.627055:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830219.627059:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830219.627061:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280984832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830220.647687:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830221.671377:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830222.694406:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830223.718145:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830224.742207:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830224.742218:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830224.742228:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830224.742238:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4774 00000100:00080000:2.0:1787830224.742248:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830224.742256:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830224.742289:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671280984960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830224.742316:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830224.742319:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830224.742321:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830224.742325:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4774 00000100:00080000:2.0:1787830224.742328:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830224.742333:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830224.742342:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830224.742355:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671280985088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830224.742364:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830224.742367:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830224.742368:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830224.742372:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4774 00000100:00080000:2.0:1787830224.742375:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830224.742378:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830224.742384:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830224.742406:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671280985216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830224.742416:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830224.742419:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830224.742421:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830224.742424:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4774 00000100:00080000:2.0:1787830224.742427:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830224.742431:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830224.742436:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830224.742446:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671280985344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830224.742454:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830224.742457:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00010000:00010000:3.0:1787830224.742715:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830224.743243:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830224.743253:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4774 00000100:00080000:2.0:1787830224.743260:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830224.743265:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830224.743273:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830224.743275:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280984960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830224.743305:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671280985472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830224.743338:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830224.743380:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830224/real 1787830224] req@ffff8ab3f043b480 x1874671280984960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830224.743396:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671280984960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830224.743406:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830224.743410:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830224.743414:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280984960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830224.743420:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280985088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830224.743447:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830224.743459:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830224/real 1787830224] req@ffff8ab3f043b840 x1874671280985088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830224.743470:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671280985088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830224.743483:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830224.743486:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830224.743489:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280985088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830224.743495:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280985216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830224.743514:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830224.743524:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830224/real 1787830224] req@ffff8ab3f0439e00 x1874671280985216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830224.743534:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671280985216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830224.743542:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830224.743546:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830224.743549:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280985216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830224.743556:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280985344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830224.743577:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830224.743591:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830224/real 1787830224] req@ffff8ab3f043a1c0 x1874671280985344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830224.743605:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671280985344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830224.743627:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830224.743631:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830224.743635:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280985344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830224.743661:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280985472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830224.743687:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830224.743702:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830224/real 1787830224] req@ffff8ab3f043a940 x1874671280985472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830224.743713:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671280985472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830224.743722:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830224.743725:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830224.743728:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280985472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830225.766316:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830226.791074:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830227.814132:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830228.839066:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830229.862250:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830229.862316:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830229.862333:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830229.862336:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830229.862346:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4779 00000100:00080000:0.0:1787830229.862357:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830229.862363:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830229.862404:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280985600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830229.862432:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830229.862435:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830229.862437:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830229.862441:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4779 00000100:00080000:0.0:1787830229.862445:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830229.862449:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830229.862458:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830229.862471:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280985728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830229.862481:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280985600:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830229.862483:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830229.862490:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830229.862492:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830229.862496:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4779 00000100:00080000:0.0:1787830229.862499:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830229.862503:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830229.862507:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830229.862518:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280985856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830229.862527:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830229.862530:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830229.862532:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830229.862536:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4779 00000100:00080000:0.0:1787830229.862539:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830229.862543:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:3.0:1787830229.862548:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787830229.862549:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830229.862562:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280985984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830229.862570:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830229.862574:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830229.862575:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830229.862579:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4779 00000100:00080000:0.0:1787830229.862581:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830229.862586:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830229.863042:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830229/real 1787830229] req@ffff8ab3f075cf00 x1874671280985600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830229.863071:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280985600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830229.863080:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830229.863084:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830229.863088:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280985600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830229.863120:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280985728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830229.863146:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830229.863160:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830229/real 1787830229] req@ffff8ab3f075e940 x1874671280985728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830229.863172:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280985728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830229.863180:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830229.863183:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830229.863186:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280985728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830229.863192:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280985856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830229.863210:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830229.863221:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830229/real 1787830229] req@ffff8ab3f075de00 x1874671280985856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830229.863232:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280985856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830229.863246:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830229.863250:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830229.863253:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280985856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830229.863259:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280985984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830229.863278:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830229.863290:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830229/real 1787830229] req@ffff8ab3f075c000 x1874671280985984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830229.863314:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280985984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830229.863324:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830229.863328:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830229.863331:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280985984:192.168.203.124@tcp:8:kworker.0 00000008:00100000:0.0:1787830229.863395:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830229.863411:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280986112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830229.867088:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280986112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830229.867154:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830229.867182:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830229/real 1787830229] req@ffff8ab3f075cb40 x1874671280986112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830229.867199:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280986112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830229.867212:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830229.867216:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830229.867221:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280986112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830230.886108:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830231.910096:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830232.938958:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830233.958124:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830234.982209:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830234.982218:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830234.982222:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830234.982232:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4784 00000100:00080000:0.0:1787830234.982242:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830234.982249:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830234.982285:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280986240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830234.982311:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830234.982314:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830234.982315:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830234.982318:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4784 00000100:00080000:0.0:1787830234.982322:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830234.982325:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830234.982416:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280986240:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787830234.982464:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787830234.982515:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830234.982525:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280986368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830234.982545:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830234.982549:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830234.982552:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830234.982556:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4784 00000100:00080000:0.0:1787830234.982561:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830234.982567:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830234.982574:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830234.982589:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280986496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830234.982595:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830234/real 1787830234] req@ffff8ab3f075f0c0 x1874671280986240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830234.982610:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830234.982617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787830234.982638:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280986240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830234.982658:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830234.982664:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830234.982668:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280986240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830234.982699:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280986368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830234.982725:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830234.982738:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830234/real 1787830234] req@ffff8ab3f075f840 x1874671280986368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830234.982752:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280986368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830234.982760:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00010000:00010000:2.0:1787830234.982765:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830234.982765:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830234.982775:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280986368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830234.982784:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280986496:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787830234.982803:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787830234.982804:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830234.982818:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4784 00000100:00100000:3.0:1787830234.982824:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830234/real 1787830234] req@ffff8ab3f075e580 x1874671280986496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830234.982825:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830234.982837:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830234.982849:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830234.982864:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280986496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830234.982866:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280986624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830234.982916:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830234.982919:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830234.982921:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830234.982928:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4784 00000100:00080000:0.0:1787830234.982932:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830234.982938:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:3.0:1787830234.982955:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787830234.982956:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:3.0:1787830234.982970:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830234.982976:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280986752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830234.982982:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280986496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830234.983023:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280986624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830234.983123:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830234.983136:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830234/real 1787830234] req@ffff8ab3f075f480 x1874671280986624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830234.983148:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280986624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830234.983162:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830234.983166:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830234.983170:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280986624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830234.983176:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280986752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830234.983197:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830234.983207:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830234/real 1787830234] req@ffff8ab3f075ed00 x1874671280986752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830234.983225:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280986752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830234.983235:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830234.983238:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830234.983241:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280986752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830236.006278:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830237.030458:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830238.054114:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830239.081248:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830240.102072:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830240.102083:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830240.102088:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830240.102098:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4789 00000100:00080000:0.0:1787830240.102108:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830240.102116:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830240.102153:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280986880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830240.102181:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1787830240.102181:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830240.102187:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830240.102189:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830240.102194:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4789 00000100:00080000:0.0:1787830240.102198:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830240.102203:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830240.102218:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830240.102224:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280986880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830240.102233:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280987008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830240.102254:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830240.102260:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830240.102262:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830240.102266:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4789 00000800:00000100:1.0:1787830240.102286:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830240.102359:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830240.102367:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830240.102373:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830240.102389:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280987136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830240.102399:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830240.102400:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830240/real 1787830240] req@ffff8ab3f075da40 x1874671280986880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830240.102406:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830240.102409:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830240.102418:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4789 00000100:00080000:0.0:1787830240.102422:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830240.102427:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830240.102432:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280986880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787830240.102434:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830240.102449:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280987264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830240.102449:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830240.102456:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830240.102459:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280986880:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830240.102475:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830240.102479:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830240.102481:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830240.102486:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280987008:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787830240.102489:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4789 00000100:00080000:0.0:1787830240.102495:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830240.102501:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830240.102506:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787830240.102517:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830240.102528:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280987392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830240.102535:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830240/real 1787830240] req@ffff8ab3f075ed00 x1874671280987008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830240.102552:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280987008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830240.102561:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830240.102565:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830240.102569:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280987008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830240.102575:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280987136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830240.102593:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830240.102603:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830240/real 1787830240] req@ffff8ab3f075f480 x1874671280987136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830240.102614:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280987136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830240.102620:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830240.102623:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830240.102626:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280987136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830240.102631:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280987264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830240.102649:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830240.102658:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830240/real 1787830240] req@ffff8ab3f075e580 x1874671280987264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830240.102669:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280987264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830240.102678:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830240.102681:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830240.102684:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280987264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830240.102711:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280987392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830240.102729:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830240.102739:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830240/real 1787830240] req@ffff8ab3f075f840 x1874671280987392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830240.102750:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280987392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830240.102759:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830240.102762:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830240.102765:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280987392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830241.128372:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830242.153351:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830243.174104:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830244.198458:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830245.222013:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830245.222024:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830245.222028:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830245.222038:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4794 00000100:00080000:0.0:1787830245.222048:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830245.222055:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830245.222090:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280987520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830245.222118:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830245.222122:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830245.222124:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830245.222128:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4794 00000100:00080000:0.0:1787830245.222132:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830245.222136:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830245.222145:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830245.222158:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280987648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830245.222168:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830245.222170:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830245.222172:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830245.222176:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4794 00000100:00080000:0.0:1787830245.222180:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830245.222184:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830245.222189:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830245.222200:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280987776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830245.222211:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830245.222214:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830245.222216:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830245.222221:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4794 00000100:00080000:0.0:1787830245.222226:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830245.222231:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830245.222237:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00010000:00010000:3.0:1787830245.222244:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830245.222250:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280987904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830245.222261:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830245.222265:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830245.222268:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830245.222271:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4794 00000100:00080000:0.0:1787830245.222275:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830245.222279:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830245.222283:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830245.222293:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280988032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830245.222806:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280987520:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830245.222861:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830245.222930:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830245/real 1787830245] req@ffff8ab3f075f0c0 x1874671280987520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830245.222947:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280987520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830245.222966:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830245.222971:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830245.222975:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280987520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830245.222983:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280987648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830245.223005:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830245.223019:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830245/real 1787830245] req@ffff8ab3f075cb40 x1874671280987648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830245.223031:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280987648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830245.223039:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830245.223042:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830245.223045:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280987648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830245.223051:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280987776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830245.223070:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830245.223080:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830245/real 1787830245] req@ffff8ab3f075c000 x1874671280987776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830245.223090:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280987776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830245.223098:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830245.223102:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830245.223106:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280987776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830245.223111:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280987904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830245.223129:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830245.223141:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830245/real 1787830245] req@ffff8ab3f075de00 x1874671280987904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830245.223152:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280987904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830245.223161:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830245.223164:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830245.223166:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280987904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830245.223172:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280988032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830245.223189:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830245.223199:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830245/real 1787830245] req@ffff8ab3f075e940 x1874671280988032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830245.223210:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280988032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830245.223219:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830245.223223:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830245.223227:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280988032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830246.246189:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830247.270193:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830248.294165:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830249.318148:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830250.341997:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830250.342008:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830250.342013:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830250.342023:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4799 00000100:00080000:0.0:1787830250.342033:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830250.342041:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830250.342079:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280988160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830250.342117:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830250.342121:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830250.342124:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830250.342128:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4799 00000100:00080000:0.0:1787830250.342133:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830250.342137:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830250.342146:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830250.342161:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280988288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830250.342170:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830250.342173:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830250.342175:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830250.342179:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4799 00000100:00080000:0.0:1787830250.342182:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830250.342185:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830250.342190:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830250.342201:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280988416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830250.342210:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830250.342213:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830250.342215:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830250.342219:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4799 00000100:00080000:0.0:1787830250.342222:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830250.342226:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830250.342230:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830250.342240:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280988544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830250.342248:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830250.342251:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830250.342252:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830250.342256:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4799 00000100:00080000:0.0:1787830250.342259:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830250.342262:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830250.342266:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830250.342275:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280988672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787830250.342306:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830250.343813:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280988160:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830250.344404:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830250.344459:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830250/real 1787830250] req@ffff8ab3f075cf00 x1874671280988160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830250.344478:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280988160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830250.344488:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830250.344493:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830250.344496:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280988160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830250.344505:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280988288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830250.344956:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830250.344975:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830250/real 1787830250] req@ffff8ab3f075e940 x1874671280988288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830250.344989:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280988288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830250.344998:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830250.345002:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830250.345005:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280988288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830250.345013:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280988416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830250.345035:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830250.345046:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830250/real 1787830250] req@ffff8ab3f075de00 x1874671280988416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830250.345057:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280988416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830250.345064:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830250.345068:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830250.345071:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280988416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830250.345078:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280988544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830250.345099:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830250.345109:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830250/real 1787830250] req@ffff8ab3f075c000 x1874671280988544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830250.345125:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280988544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830250.345135:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830250.345138:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830250.345141:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280988544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830250.345147:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280988672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830250.345175:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830250.345186:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830250/real 1787830250] req@ffff8ab3f075cb40 x1874671280988672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830250.345197:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280988672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830250.345206:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830250.345209:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830250.345212:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280988672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830251.366239:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830252.390125:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830253.414208:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830254.438137:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830255.462669:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830255.462680:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830255.462685:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830255.462696:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4805 00000100:00080000:0.0:1787830255.462706:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830255.462714:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830255.462754:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280988800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830255.462790:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830255.462793:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830255.462795:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830255.462799:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4805 00000100:00080000:0.0:1787830255.462802:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830255.462940:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280988800:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830255.463041:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830255.463053:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830255.463067:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280988928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830255.463078:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787830255.463195:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830255.463243:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830255/real 1787830255] req@ffff8ab3f075f0c0 x1874671280988800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830255.463259:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280988800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830255.463269:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830255.463273:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830255.463276:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280988800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830255.463301:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280988928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830255.463324:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830255.463335:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830255/real 1787830255] req@ffff8ab3f075f840 x1874671280988928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830255.463346:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280988928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830255.463354:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830255.463357:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830255.463360:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280988928:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787830255.463661:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830255.463663:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830255.463669:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4805 00000100:00080000:0.0:1787830255.463674:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830255.463678:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830255.463684:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830255.463699:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280989056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830255.464057:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280989056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830255.464086:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830255.464123:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830255/real 1787830255] req@ffff8ab3f075e580 x1874671280989056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830255.464129:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830255.464137:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830255.464139:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830255.464143:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280989056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830255.464147:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4805 00000100:00080000:0.0:1787830255.464152:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830255.464158:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:3.0:1787830255.464167:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787830255.464169:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:3.0:1787830255.464173:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830255.464179:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280989056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830255.464188:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280989184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830255.464216:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830255.464220:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830255.464223:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830255.464227:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4805 00000100:00080000:0.0:1787830255.464231:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830255.464235:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280989184:192.168.203.124@tcp:8:kworker.0 00000100:00080000:0.0:1787830255.464238:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830255.464245:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830255.464260:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280989312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830255.464270:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830255.464285:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830255/real 1787830255] req@ffff8ab3f075f480 x1874671280989184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830255.464298:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280989184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830255.464316:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830255.464319:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830255.464322:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280989184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830255.464332:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280989312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830255.464347:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830255.464355:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830255/real 1787830255] req@ffff8ab3f075ed00 x1874671280989312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830255.464542:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280989312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830255.464551:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830255.464554:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830255.464557:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280989312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830255.465967:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830256.486813:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830257.510162:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830258.535160:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830259.558229:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830260.582157:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830260.582166:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830260.582170:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830260.582180:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4810 00000100:00080000:1.0:1787830260.582190:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830260.582198:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830260.582240:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280989440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830260.582264:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830260.582267:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830260.582269:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830260.582288:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4810 00000100:00080000:1.0:1787830260.582293:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830260.582298:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830260.582307:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280989440:192.168.203.124@tcp:250:kworker.0 00000008:00100000:1.0:1787830260.582308:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830260.582324:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280989568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830260.582332:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830260.582334:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830260.582336:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830260.582339:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4810 00000100:00080000:1.0:1787830260.582342:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830260.582346:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830260.582351:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830260.582364:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280989696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830260.582374:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830260.582377:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830260.582381:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787830260.582382:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:1.0:1787830260.582390:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4810 00000100:00080000:1.0:1787830260.582394:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830260.582407:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830260.582412:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830260.582425:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280989824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830260.582433:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830260.582437:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830260.582439:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830260.582442:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830260/real 1787830260] req@ffff8ab3d22fed00 x1874671280989440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830260.582445:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4810 00000100:00080000:1.0:1787830260.582450:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830260.582457:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830260.582463:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830260.582468:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280989440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830260.582476:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280989952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830260.582482:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830260.582489:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830260.582493:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280989440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830260.582522:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280989568:192.168.203.124@tcp:38:kworker.0 00010000:00010000:1.0:1787830260.582536:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1787830260.582547:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830260.582563:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830260/real 1787830260] req@ffff8ab3d22fc780 x1874671280989568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830260.582575:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280989568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830260.582583:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830260.582586:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830260.582589:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280989568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830260.582595:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280989696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830260.582613:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830260.582631:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830260/real 1787830260] req@ffff8ab3d22fd2c0 x1874671280989696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830260.582641:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280989696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830260.582649:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830260.582653:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830260.582656:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280989696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830260.582661:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280989824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830260.582680:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830260.582691:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830260/real 1787830260] req@ffff8ab3d22fe1c0 x1874671280989824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830260.582701:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280989824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830260.582709:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830260.582712:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830260.582715:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280989824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830260.582720:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280989952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830260.582737:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830260.582747:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830260/real 1787830260] req@ffff8ab3d22fde00 x1874671280989952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830260.582757:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280989952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830260.582765:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830260.582768:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830260.582770:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280989952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830261.606120:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830262.630177:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830263.654091:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830264.679146:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830265.702150:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830265.702166:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830265.702170:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830265.702183:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4815 00000100:00080000:0.0:1787830265.702194:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830265.702201:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830265.702257:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280990080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830265.702291:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830265.702295:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830265.702296:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830265.702300:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4815 00000100:00080000:0.0:1787830265.702304:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830265.702307:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830265.702317:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830265.702331:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280990208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830265.702340:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830265.702343:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830265.702344:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830265.702348:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4815 00000100:00080000:0.0:1787830265.702351:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830265.702355:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830265.702359:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830265.702369:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280990336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830265.702377:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830265.702381:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830265.702382:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830265.702386:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4815 00000100:00080000:0.0:1787830265.702389:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830265.702393:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830265.702397:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830265.702406:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280990464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830265.702414:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830265.702417:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830265.702418:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830265.702422:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4815 00000100:00080000:0.0:1787830265.702425:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830265.702429:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830265.702432:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830265.702441:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280990592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830265.702930:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280990080:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830265.702982:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830265.703023:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830265/real 1787830265] req@ffff8ab3f075da40 x1874671280990080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830265.703039:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280990080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830265.703049:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830265.703052:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830265.703055:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280990080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830265.703240:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280990208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830265.703270:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830265.703283:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830265/real 1787830265] req@ffff8ab3f075ed00 x1874671280990208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830265.703345:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280990208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830265.703353:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830265.703356:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830265.703360:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280990208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830265.703421:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280990336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830265.703443:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830265.703455:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830265/real 1787830265] req@ffff8ab3f075f480 x1874671280990336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830265.703465:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280990336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830265.703478:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830265.703482:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830265.703486:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280990336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830265.703493:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280990464:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830265.703499:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1787830265.703525:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830265.703538:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830265/real 1787830265] req@ffff8ab3f075e580 x1874671280990464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830265.703549:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280990464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830265.703558:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830265.703562:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830265.703565:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280990464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830265.703570:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280990592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830265.703588:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830265.703598:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830265/real 1787830265] req@ffff8ab3f075f840 x1874671280990592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830265.703610:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280990592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830265.703618:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830265.703622:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830265.703625:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280990592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830266.726059:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830267.750122:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830268.774956:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830269.802036:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830270.822026:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830270.822036:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830270.822040:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830270.822050:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4820 00010000:00010000:1.0:1787830270.822052:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830270.822062:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830270.822070:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830270.822108:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280990720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830270.822133:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830270.822136:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830270.822138:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830270.822142:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4820 00000100:00080000:0.0:1787830270.822145:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830270.822149:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830270.822158:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830270.822171:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280990848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830270.822177:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280990720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830270.822187:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830270.822191:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830270.822193:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830270.822198:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4820 00000100:00080000:0.0:1787830270.822202:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830270.822206:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830270.822211:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830270.822222:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280990976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830270.822232:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830270.822235:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830270.822237:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830270.822241:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4820 00000100:00080000:0.0:1787830270.822245:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787830270.822247:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830270.822253:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830270.822259:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830270.822270:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280991104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830270.822279:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830270.822282:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830270.822285:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830270.822289:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4820 00000100:00080000:0.0:1787830270.822292:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830270.822296:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830270.822302:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830270.822302:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830270/real 1787830270] req@ffff8ab3f075f0c0 x1874671280990720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830270.822321:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280991232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830270.822327:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280990720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830270.822343:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830270.822347:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830270.822352:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280990720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830270.822387:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280990848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830270.822409:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830270.822421:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830270/real 1787830270] req@ffff8ab3f075cb40 x1874671280990848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830270.822432:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280990848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830270.822440:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830270.822443:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830270.822447:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280990848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830270.822452:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280990976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830270.822470:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830270.822481:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830270/real 1787830270] req@ffff8ab3f075c000 x1874671280990976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830270.822492:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280990976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830270.822500:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830270.822503:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830270.822506:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280990976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830270.822512:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280991104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830270.822530:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830270.822540:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830270/real 1787830270] req@ffff8ab3f075de00 x1874671280991104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830270.822552:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280991104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830270.822561:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830270.822565:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830270.822568:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280991104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830270.822573:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280991232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830270.822590:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830270.822600:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830270/real 1787830270] req@ffff8ab3f075e940 x1874671280991232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830270.822611:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280991232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830270.822619:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830270.822622:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830270.822625:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280991232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830271.846311:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830272.870110:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830273.894778:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830274.918053:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830275.942054:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830275.942063:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830275.942067:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830275.942075:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4825 00000100:00080000:2.0:1787830275.942083:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830275.942088:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830275.942122:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671280991360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787830275.942125:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830275.942141:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830275.942144:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830275.942146:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830275.942148:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4825 00000100:00080000:2.0:1787830275.942151:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830275.942153:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830275.942160:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830275.942166:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280991360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830275.942170:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671280991488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830275.942188:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830275.942190:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830275.942192:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830275.942194:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4825 00000100:00080000:2.0:1787830275.942197:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830275.942199:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830275.942202:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830275.942210:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671280991616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830275.942217:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830275.942219:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830275.942221:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830275.942223:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4825 00000100:00080000:2.0:1787830275.942225:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830275.942227:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830275.942230:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:3.0:1787830275.942235:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830275.942240:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671280991744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830275.942247:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830275.942251:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830275.942253:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830275.942258:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4825 00000100:00080000:2.0:1787830275.942261:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830275.942266:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830275.942271:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830275.942283:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671280991872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830275.942288:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830275/real 1787830275] req@ffff8ab3f0439a40 x1874671280991360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830275.942314:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671280991360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830275.942324:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830275.942327:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830275.942331:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280991360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830275.942353:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280991488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830275.942373:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830275.942385:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830275/real 1787830275] req@ffff8ab3f043bc00 x1874671280991488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830275.942395:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671280991488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830275.942402:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830275.942405:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830275.942409:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280991488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830275.942414:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280991616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830275.942431:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830275.942442:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830275/real 1787830275] req@ffff8ab3f04383c0 x1874671280991616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830275.942452:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671280991616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830275.942459:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830275.942462:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830275.942465:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280991616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830275.942470:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280991744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830275.942487:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830275.942497:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830275/real 1787830275] req@ffff8ab3f043ad00 x1874671280991744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830275.942507:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671280991744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830275.942516:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830275.942519:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830275.942521:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280991744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830275.942527:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280991872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830275.942543:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830275.942553:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830275/real 1787830275] req@ffff8ab3f04392c0 x1874671280991872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830275.942562:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671280991872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830275.942570:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830275.942573:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830275.942575:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280991872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830276.966142:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830277.990392:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830279.014045:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830280.038272:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830281.062060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830281.066000:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830281.066018:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830281.066023:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830281.066033:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4830 00000100:00080000:0.0:1787830281.066044:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830281.066053:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830281.066089:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280992000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830281.066118:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830281.066125:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830281.066127:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830281.066131:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4830 00000100:00080000:0.0:1787830281.066134:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830281.066138:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830281.066147:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830281.066160:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280992128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830281.066170:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830281.066173:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830281.066174:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830281.066178:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4830 00000100:00080000:0.0:1787830281.066181:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830281.066184:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830281.066188:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830281.066198:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280992256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830281.066206:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830281.066209:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830281.066211:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830281.066215:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4830 00000100:00080000:0.0:1787830281.066218:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830281.066222:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830281.066227:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830281.066235:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280992384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830281.066243:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830281.066246:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830281.066247:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830281.066251:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4830 00000100:00080000:0.0:1787830281.066254:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830281.066257:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830281.066261:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830281.066269:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280992512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830281.066337:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280992000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830281.066435:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280992128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830281.066460:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280992256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830281.066482:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280992384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830281.066501:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280992512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830282.086193:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830283.110142:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787830284.134281:0:10942:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787830284.134364:0:10942:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787830284.134379:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787830284.134414:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787830284.134424:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787830284.134440:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:3.0:1787830284.134445:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830281/real 1787830284] req@ffff8ab3f075cf00 x1874671280992000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787830284.134452:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:3.0:1787830284.134471:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280992000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830284.134479:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830284.134482:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830284.134485:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280992000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830284.134490:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830281/real 1787830284] req@ffff8ab3f075e940 x1874671280992128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830284.134497:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280992128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830284.134501:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830284.134503:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830284.134505:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280992128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830284.134509:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830281/real 1787830284] req@ffff8ab3f075de00 x1874671280992256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830284.134515:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280992256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830284.134519:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830284.134521:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830284.134523:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280992256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830284.134526:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830281/real 1787830284] req@ffff8ab3f075c000 x1874671280992384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830284.134531:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280992384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830284.134537:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830284.134538:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830284.134540:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280992384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830284.134543:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830281/real 1787830284] req@ffff8ab3f075cb40 x1874671280992512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830284.134550:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280992512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830284.134555:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830284.134556:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830284.134558:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280992512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830284.134577:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830285.158107:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830286.182583:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830286.182591:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830286.182595:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830286.182604:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4835 00010000:00010000:2.0:1787830286.183208:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830286.183508:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830286.183520:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830286.183560:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280992640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830286.183592:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830286.183596:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830286.183598:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830286.183602:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4835 00000100:00080000:1.0:1787830286.183606:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830286.183611:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830286.183622:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830286.183635:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280992768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830286.183644:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830286.183647:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830286.183648:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830286.183653:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4835 00000100:00080000:1.0:1787830286.183656:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830286.183660:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830286.183665:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830286.183687:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280992896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830286.183697:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830286.183701:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830286.183703:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830286.183706:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4835 00000100:00080000:1.0:1787830286.183710:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830286.183715:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830286.183721:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280992640:192.168.203.124@tcp:250:kworker.0 00000008:00100000:1.0:1787830286.183722:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830286.183749:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280993024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830286.183760:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830286.183763:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830286.183764:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830286.183768:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4835 00000100:00080000:1.0:1787830286.183781:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830286.183784:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830286.183789:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:3.0:1787830286.183795:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830286.183805:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280993152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830286.183849:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830286/real 1787830286] req@ffff8ab3d22fde00 x1874671280992640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830286.183867:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280992640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830286.183878:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830286.183882:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830286.183905:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280992640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830286.183913:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280992768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830286.183937:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830286.183949:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830286/real 1787830286] req@ffff8ab3d22fe1c0 x1874671280992768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830286.183961:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280992768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830286.183970:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830286.183974:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830286.183978:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280992768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830286.183984:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280992896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830286.184002:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830286.184013:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830286/real 1787830286] req@ffff8ab3d22fd2c0 x1874671280992896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830286.184023:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280992896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830286.184030:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830286.184033:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830286.184036:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280992896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830286.184065:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280993024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830286.184086:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830286.184098:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830286/real 1787830286] req@ffff8ab3d22fc780 x1874671280993024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830286.184110:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280993024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830286.184121:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830286.184124:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830286.184127:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280993024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830286.184133:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280993152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830286.184151:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830286.184162:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830286/real 1787830286] req@ffff8ab3d22fed00 x1874671280993152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830286.184174:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280993152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830286.184183:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830286.184186:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830286.184189:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280993152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830287.206354:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830288.230097:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830289.255148:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830290.278120:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830291.302207:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830291.302217:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830291.302222:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830291.302232:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4840 00000100:00080000:0.0:1787830291.302243:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830291.302251:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830291.302296:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280993280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830291.302322:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830291.302325:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830291.302327:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830291.302330:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4840 00000100:00080000:0.0:1787830291.302334:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830291.302337:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830291.302346:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830291.302356:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280993408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830291.302365:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830291.302368:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830291.302369:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830291.302373:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4840 00000100:00080000:0.0:1787830291.302375:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830291.302378:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830291.302382:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830291.302391:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280993536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830291.302399:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830291.302402:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830291.302404:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830291.302408:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4840 00000100:00080000:0.0:1787830291.302410:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830291.302414:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830291.302418:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830291.302428:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280993664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830291.302437:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830291.302440:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830291.302442:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830291.302445:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4840 00000100:00080000:0.0:1787830291.302448:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830291.302452:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830291.302456:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830291.302466:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280993792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830291.302869:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280993280:192.168.203.124@tcp:250:kworker.0 00010000:00010000:2.0:1787830291.303022:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1787830291.303024:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830291.303141:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830291/real 1787830291] req@ffff8ab3f075f0c0 x1874671280993280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830291.303158:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280993280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830291.303169:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830291.303173:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830291.303177:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280993280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830291.303184:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280993408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830291.303206:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830291.303219:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830291/real 1787830291] req@ffff8ab3f075f840 x1874671280993408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830291.303230:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280993408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830291.303238:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830291.303241:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830291.303244:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280993408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830291.303249:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280993536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830291.303264:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830291.303273:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830291/real 1787830291] req@ffff8ab3f075e580 x1874671280993536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830291.303283:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280993536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830291.303290:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830291.303293:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830291.303296:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280993536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830291.303301:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280993664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830291.303315:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830291.303324:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830291/real 1787830291] req@ffff8ab3f075f480 x1874671280993664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830291.303333:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280993664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830291.303342:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830291.303345:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830291.303348:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280993664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830291.303353:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280993792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830291.303367:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830291.303376:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830291/real 1787830291] req@ffff8ab3f075ed00 x1874671280993792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830291.303385:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280993792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830291.303393:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830291.303396:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830291.303398:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280993792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830292.326647:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830293.350127:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830294.375531:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830295.398181:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830296.422011:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830296.422022:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830296.422026:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830296.422036:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4846 00000100:00080000:0.0:1787830296.422047:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830296.422053:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787830296.422067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830296.422092:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671280993920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830296.422129:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830296.422133:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830296.422135:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830296.422139:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4846 00000100:00080000:0.0:1787830296.422142:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830296.422146:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830296.422155:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830296.422169:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280994048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830296.422177:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830296.422181:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830296.422182:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830296.422186:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4846 00000100:00080000:0.0:1787830296.422189:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830296.422193:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830296.422197:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830296.422207:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280994176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830296.422215:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830296.422218:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830296.422220:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830296.422223:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4846 00000100:00080000:0.0:1787830296.422226:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830296.422231:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830296.422235:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830296.422243:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280994304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830296.422251:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830296.422254:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830296.422255:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830296.422259:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4846 00000100:00080000:0.0:1787830296.422261:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830296.422265:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830296.422269:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830296.422278:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280994432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830296.422652:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280993920:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830296.422714:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830296.422754:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830296/real 1787830296] req@ffff8ab3f075da40 x1874671280993920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830296.423334:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671280993920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830296.423346:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830296.423350:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830296.423354:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280993920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830296.423362:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280994048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830296.423388:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830296.423405:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830296/real 1787830296] req@ffff8ab3f075ed00 x1874671280994048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830296.423418:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280994048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830296.423426:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830296.423430:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830296.423434:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280994048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830296.423441:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280994176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830296.423480:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830296.423491:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830296/real 1787830296] req@ffff8ab3f075f480 x1874671280994176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830296.423501:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280994176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830296.423509:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830296.423511:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830296.423514:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280994176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830296.423520:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280994304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830296.423543:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830296.423554:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830296/real 1787830296] req@ffff8ab3f075e580 x1874671280994304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830296.423564:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280994304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830296.423573:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830296.423576:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830296.423579:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280994304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830296.423585:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280994432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830296.423615:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830296.423626:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830296/real 1787830296] req@ffff8ab3f075f840 x1874671280994432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830296.423636:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280994432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830296.423645:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830296.423648:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830296.423651:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280994432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830297.449738:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830298.470345:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830299.496824:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830300.518148:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830301.542019:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830301.542031:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830301.542036:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830301.542047:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4851 00000100:00080000:0.0:1787830301.542057:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830301.542064:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830301.542099:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280994560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787830301.542108:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830301.542132:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830301.542138:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830301.542140:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830301.542143:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4851 00000100:00080000:0.0:1787830301.542146:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830301.542150:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830301.542159:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830301.542169:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280994688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830301.542178:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830301.542181:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830301.542184:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830301.542189:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4851 00000100:00080000:0.0:1787830301.542192:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830301.542197:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830301.542202:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830301.542211:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280994560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830301.542220:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280994816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830301.542231:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830301.542234:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830301.542236:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830301.542241:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4851 00000100:00080000:0.0:1787830301.542244:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830301.542249:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830301.542255:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830301.542265:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280994944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830301.542276:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830301.542277:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830301.542281:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830301.542283:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830301.542289:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4851 00000100:00080000:0.0:1787830301.542293:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830301.542297:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830301.542301:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830301.542312:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280995072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830301.542339:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830301/real 1787830301] req@ffff8ab3f075f0c0 x1874671280994560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830301.542358:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280994560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830301.542368:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830301.542371:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830301.542375:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280994560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830301.542381:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280994688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830301.542402:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830301.542414:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830301/real 1787830301] req@ffff8ab3f075cb40 x1874671280994688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830301.542423:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280994688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830301.542431:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830301.542433:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830301.542435:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280994688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830301.542456:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280994816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830301.542474:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830301.542483:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830301/real 1787830301] req@ffff8ab3f075c000 x1874671280994816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830301.542493:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280994816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830301.542500:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830301.542503:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830301.542506:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280994816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830301.542512:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280994944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830301.542530:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830301.542540:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830301/real 1787830301] req@ffff8ab3f075de00 x1874671280994944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830301.542551:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280994944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830301.542560:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830301.542574:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830301.542578:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280994944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830301.542583:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280995072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830301.542601:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830301.542611:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830301/real 1787830301] req@ffff8ab3f075e940 x1874671280995072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830301.542621:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280995072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830301.542630:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830301.542633:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830301.542635:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280995072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830302.566443:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830303.590689:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830304.614184:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830305.638459:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830306.662033:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830306.662043:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830306.662047:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830306.662057:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4856 00000100:00080000:0.0:1787830306.662066:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830306.662073:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787830306.662086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830306.662109:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671280995200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830306.662132:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830306.662136:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830306.662137:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830306.662140:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4856 00000100:00080000:0.0:1787830306.662148:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830306.662151:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830306.662161:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830306.662174:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671280995328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830306.662184:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830306.662184:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280995200:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830306.662192:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830306.662194:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830306.662198:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4856 00000100:00080000:0.0:1787830306.662201:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830306.662204:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830306.662208:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830306.662219:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671280995456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830306.662228:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830306.662231:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830306.662232:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830306.662236:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4856 00000100:00080000:0.0:1787830306.662246:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830306.662250:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830306.662254:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:3.0:1787830306.662263:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830306.662265:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671280995584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830306.662281:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830306.662285:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830306.662287:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830306.662290:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4856 00000100:00080000:0.0:1787830306.662293:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830306.662297:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830306.662301:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830306.662313:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671280995712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830306.662317:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830306/real 1787830306] req@ffff8ab3f075cf00 x1874671280995200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830306.662343:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671280995200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830306.662363:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830306.662367:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830306.662370:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280995200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830306.662392:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280995328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830306.662414:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830306.662425:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830306/real 1787830306] req@ffff8ab3f075e940 x1874671280995328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830306.662436:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671280995328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830306.662444:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830306.662447:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830306.662450:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280995328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830306.662456:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280995456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830306.662474:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830306.662485:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830306/real 1787830306] req@ffff8ab3f075de00 x1874671280995456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830306.662497:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671280995456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830306.662511:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830306.662514:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830306.662518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280995456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830306.662524:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280995584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830306.662564:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830306.662575:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830306/real 1787830306] req@ffff8ab3f075c000 x1874671280995584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830306.662587:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671280995584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830306.662597:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830306.662600:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830306.662603:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280995584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830306.662608:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280995712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830306.662622:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830306.662629:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830306/real 1787830306] req@ffff8ab3f075cb40 x1874671280995712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830306.662640:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671280995712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830306.662647:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830306.662650:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830306.662653:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280995712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830307.686195:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830308.710760:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830309.734542:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830310.758197:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830311.782099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830311.782110:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830311.782114:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830311.782124:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4861 00000100:00080000:2.0:1787830311.782136:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830311.782146:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830311.782183:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671280995840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830311.782203:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830311.782207:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830311.782210:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830311.782214:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4861 00000100:00080000:2.0:1787830311.782217:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830311.782221:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830311.782230:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830311.782243:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671280995968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830311.782253:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830311.782255:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830311.782257:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830311.782261:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4861 00000100:00080000:2.0:1787830311.782264:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830311.782268:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830311.782272:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830311.782280:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671280996096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830311.782288:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830311.782290:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830311.782292:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830311.782295:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4861 00000100:00080000:2.0:1787830311.782297:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830311.782301:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830311.782306:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830311.782314:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671280996224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830311.782322:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830311.782325:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830311.782327:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830311.782331:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4861 00000100:00080000:2.0:1787830311.782334:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830311.782337:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830311.782341:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830311.782351:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671280996352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787830311.782586:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830311.782620:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280995840:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830311.782701:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830311.782751:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830311/real 1787830311] req@ffff8ab3f0439680 x1874671280995840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830311.782765:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671280995840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830311.782775:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830311.782778:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830311.782781:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280995840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830311.782788:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280995968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830311.782810:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830311.782821:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830311/real 1787830311] req@ffff8ab3f043b0c0 x1874671280995968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830311.782833:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671280995968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830311.782842:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830311.782846:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830311.782849:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280995968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830311.782856:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280996096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830311.782873:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830311.782880:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830311/real 1787830311] req@ffff8ab3f043a580 x1874671280996096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830311.782919:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671280996096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830311.782925:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830311.782927:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830311.782930:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280996096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830311.782936:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280996224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830311.782955:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830311.782963:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830311/real 1787830311] req@ffff8ab3f0438f00 x1874671280996224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830311.782970:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671280996224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830311.782977:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830311.782979:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830311.782981:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280996224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830311.782985:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280996352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830311.782996:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830311.783004:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830311/real 1787830311] req@ffff8ab3f0438b40 x1874671280996352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830311.783021:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671280996352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830311.783032:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830311.783035:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830311.783038:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280996352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830312.806553:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830313.831800:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830314.854099:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830315.878331:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830316.902133:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830316.902302:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830316.902309:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830316.902313:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830316.902324:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4866 00000100:00080000:1.0:1787830316.902334:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830316.902341:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830316.902377:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280996480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830316.902401:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830316.902404:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830316.902406:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830316.902409:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4866 00000100:00080000:1.0:1787830316.902413:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830316.902417:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830316.902426:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830316.902437:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280996608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830316.902445:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830316.902448:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830316.902450:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830316.902454:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4866 00000100:00080000:1.0:1787830316.902471:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830316.902475:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830316.902480:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830316.902491:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280996736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830316.902506:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280996480:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787830316.902511:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830316.902515:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830316.902517:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830316.902521:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4866 00000100:00080000:1.0:1787830316.902525:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830316.902528:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830316.902534:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830316.902546:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280996864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830316.902554:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830316.902557:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830316.902559:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830316.902563:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4866 00000100:00080000:1.0:1787830316.902568:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830316.902572:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830316.902577:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:3.0:1787830316.902581:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830316.902591:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280996992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830316.902680:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830316/real 1787830316] req@ffff8ab3d22fd680 x1874671280996480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830316.902699:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280996480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830316.902710:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830316.902714:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830316.902718:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280996480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830316.902726:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280996608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830316.902749:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830316.902761:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830316/real 1787830316] req@ffff8ab3d22fcb40 x1874671280996608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830316.902863:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280996608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830316.902871:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830316.902875:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830316.902877:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280996608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830316.902928:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280996736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830316.902950:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830316.902962:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830316/real 1787830316] req@ffff8ab3d22fda40 x1874671280996736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830316.902973:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280996736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830316.902981:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830316.902984:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830316.902987:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280996736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830316.902993:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280996864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830316.903012:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830316.903022:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830316/real 1787830316] req@ffff8ab3d22fc000 x1874671280996864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830316.903033:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280996864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830316.903048:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830316.903051:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830316.903054:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280996864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830316.903060:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280996992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830316.903078:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830316.903088:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830316/real 1787830316] req@ffff8ab3d22ff480 x1874671280996992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830316.903099:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280996992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830316.903108:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830316.903111:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830316.903114:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280996992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830317.926570:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830318.950689:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830319.974914:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830320.998265:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830322.022258:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830322.022267:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830322.022272:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830322.022281:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4871 00000100:00080000:3.0:1787830322.022291:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830322.022297:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830322.022331:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671280997120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830322.022366:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830322.022370:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830322.022371:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830322.022430:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4871 00000100:00080000:3.0:1787830322.022435:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830322.022439:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830322.022448:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830322.022461:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671280997248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830322.022471:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280997120:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787830322.022473:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830322.022480:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830322.022482:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830322.022487:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4871 00000100:00080000:3.0:1787830322.022490:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830322.022494:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830322.022498:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830322.022510:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671280997376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830322.022519:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830322.022522:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830322.022524:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830322.022527:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4871 00000100:00080000:3.0:1787830322.022531:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830322.022536:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787830322.022538:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787830322.022542:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830322.022554:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671280997504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830322.022562:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830322.022565:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830322.022566:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830322.022570:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4871 00010000:00010000:2.0:1787830322.022625:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830322.022679:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830322.022685:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830322.022690:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830322.022700:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671280997632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830322.022711:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830322/real 1787830322] req@ffff8ab3d0a421c0 x1874671280997120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830322.022732:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671280997120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830322.022742:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830322.022745:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830322.022749:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280997120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830322.022770:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280997248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830322.022942:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830322.022955:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830322/real 1787830322] req@ffff8ab3d0a40000 x1874671280997248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830322.022964:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671280997248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830322.022969:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830322.022971:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830322.022973:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280997248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830322.022977:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280997376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830322.022993:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830322.023000:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830322/real 1787830322] req@ffff8ab3d0a43c00 x1874671280997376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830322.023007:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671280997376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830322.023012:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830322.023014:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830322.023016:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280997376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830322.023019:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280997504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830322.023028:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830322.023033:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830322/real 1787830322] req@ffff8ab3d0a41e00 x1874671280997504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830322.023039:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671280997504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830322.023048:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830322.023050:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830322.023052:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280997504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830322.023054:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280997632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830322.023063:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830322.023067:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830322/real 1787830322] req@ffff8ab3d0a430c0 x1874671280997632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830322.023073:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671280997632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830322.023079:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830322.023080:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830322.023082:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280997632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830323.048955:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830324.070187:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830325.097107:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830326.118163:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830327.142115:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830327.142212:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830327.142219:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830327.142224:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830327.142232:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4876 00000100:00080000:1.0:1787830327.142242:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830327.142248:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830327.142284:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671280997760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830327.142310:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830327.142313:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830327.142315:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830327.142319:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4876 00000100:00080000:1.0:1787830327.142322:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830327.142326:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830327.142334:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830327.142345:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671280997888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830327.142354:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830327.142358:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830327.142360:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830327.142364:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4876 00000100:00080000:1.0:1787830327.142366:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830327.142377:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830327.142382:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830327.142408:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671280998016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830327.142416:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830327.142420:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830327.142421:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830327.142425:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4876 00000100:00080000:1.0:1787830327.142428:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830327.142431:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830327.142436:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830327.142445:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671280998144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830327.142453:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830327.142456:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830327.142457:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830327.142460:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4876 00000100:00080000:1.0:1787830327.142463:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830327.142468:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830327.142471:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830327.142479:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671280998272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830327.142542:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280997760:192.168.203.124@tcp:250:kworker.0 00000800:00000100:0.0:1787830327.142599:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830327.142664:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830327/real 1787830327] req@ffff8ab3d22ff480 x1874671280997760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830327.142680:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671280997760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830327.142689:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830327.142693:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830327.142696:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280997760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830327.142703:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280997888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830327.142721:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830327.142733:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830327/real 1787830327] req@ffff8ab3d22fc000 x1874671280997888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830327.142742:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671280997888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830327.142749:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830327.142752:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830327.142755:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280997888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830327.142759:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280998016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830327.142774:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830327.142784:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830327/real 1787830327] req@ffff8ab3d22fda40 x1874671280998016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830327.142793:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671280998016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830327.142800:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830327.142802:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830327.142805:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280998016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830327.142810:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280998144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830327.142828:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830327.142837:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830327/real 1787830327] req@ffff8ab3d22fcb40 x1874671280998144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830327.142847:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671280998144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830327.142855:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830327.142858:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830327.142861:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280998144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830327.142866:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280998272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830327.142883:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830327.142916:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830327/real 1787830327] req@ffff8ab3d22fd680 x1874671280998272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830327.142927:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671280998272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830327.142935:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830327.142937:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830327.142940:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280998272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830328.166234:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830329.190123:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830330.214220:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830331.239431:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830332.262035:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830332.262045:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830332.262050:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830332.262060:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4881 00000100:00080000:1.0:1787830332.262070:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830332.262078:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830332.262114:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671280998400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830332.262136:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830332.262139:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830332.262141:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830332.262145:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4881 00000100:00080000:1.0:1787830332.262148:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830332.262151:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830332.262160:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830332.262172:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671280998528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830332.262181:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830332.262184:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830332.262186:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830332.262251:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4881 00000100:00080000:1.0:1787830332.262256:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830332.262261:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830332.262266:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830332.262280:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671280998656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830332.262290:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830332.262293:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830332.262295:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830332.262299:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4881 00000100:00080000:1.0:1787830332.262303:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830332.262307:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830332.262312:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830332.262324:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671280998784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830332.262333:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830332.262336:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830332.262337:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830332.262341:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4881 00000100:00080000:1.0:1787830332.262344:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830332.262357:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830332.262363:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830332.262375:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671280998912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830332.262451:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280998400:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830332.262509:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830332.262548:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830332/real 1787830332] req@ffff8ab3d22fed00 x1874671280998400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830332.262563:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671280998400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830332.262573:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830332.262577:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830332.262581:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280998400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830332.262590:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280998528:192.168.203.124@tcp:38:kworker.0 00010000:00010000:2.0:1787830332.262606:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:1.0:1787830332.262612:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830332.262628:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830332/real 1787830332] req@ffff8ab3d22fc780 x1874671280998528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830332.262642:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671280998528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830332.262650:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830332.262654:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830332.262657:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280998528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830332.262663:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280998656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830332.262683:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830332.262695:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830332/real 1787830332] req@ffff8ab3d22fd2c0 x1874671280998656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830332.262708:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671280998656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830332.262716:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830332.262719:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830332.262723:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280998656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830332.262730:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280998784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830332.262767:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830332.262781:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830332/real 1787830332] req@ffff8ab3d22fe1c0 x1874671280998784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830332.262792:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671280998784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830332.262801:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830332.262803:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830332.262806:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280998784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830332.262811:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280998912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830332.262828:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830332.262838:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830332/real 1787830332] req@ffff8ab3d22fde00 x1874671280998912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830332.262848:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671280998912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830332.262856:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830332.262859:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830332.262862:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280998912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830333.286209:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830334.310190:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830335.334232:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830336.358374:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830337.383089:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830337.383123:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830337.383132:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830337.383135:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830337.383145:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4887 00000100:00080000:0.0:1787830337.383155:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830337.383161:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830337.383197:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671280999040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830337.383250:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280999040:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830337.383306:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830337.383346:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830337.383349:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830337.383351:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830337.383355:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4887 00000100:00080000:0.0:1787830337.383359:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830337.383364:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830337.383373:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830337.383387:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671280999168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830337.383449:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830337/real 1787830337] req@ffff8ab3f075f0c0 x1874671280999040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830337.383466:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671280999040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830337.383476:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830337.383480:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830337.383483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280999040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830337.383505:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280999168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830337.383528:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830337.383541:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830337/real 1787830337] req@ffff8ab3f075f840 x1874671280999168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830337.383554:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671280999168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830337.383561:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830337.383564:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830337.383567:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280999168:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787830337.383613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830337.383617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830337.383619:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830337.383623:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4887 00000100:00080000:0.0:1787830337.383629:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830337.383633:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830337.383639:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830337.383652:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671280999296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830337.383663:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830337.383666:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830337.383668:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830337.383672:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4887 00000100:00080000:0.0:1787830337.383675:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830337.383679:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830337.383683:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830337.383693:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671280999424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830337.383702:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830337.383705:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830337.383706:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830337.383710:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4887 00000100:00080000:0.0:1787830337.383712:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830337.383716:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830337.383720:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830337.383729:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671280999552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830337.383820:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280999296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830337.383844:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830337.383856:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830337/real 1787830337] req@ffff8ab3f075e580 x1874671280999296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830337.383865:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671280999296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830337.383871:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830337.383872:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830337.383875:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280999296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830337.383879:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280999424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830337.383910:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830337.383917:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830337/real 1787830337] req@ffff8ab3f075f480 x1874671280999424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830337.383932:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671280999424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830337.383939:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830337.383941:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830337.383943:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280999424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830337.383946:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280999552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830337.383956:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830337.383962:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830337/real 1787830337] req@ffff8ab3f075ed00 x1874671280999552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830337.383969:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671280999552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830337.383974:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830337.383975:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830337.383977:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280999552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830338.406333:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830339.430233:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830340.454081:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830341.478188:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830342.502179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830342.502191:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830342.502195:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830342.502207:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4892 00000100:00080000:3.0:1787830342.502217:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830342.502225:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830342.502261:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671280999680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830342.502372:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280999680:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830342.502432:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830342.502465:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830342.502471:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830342.502473:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830342.502477:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4892 00000100:00080000:3.0:1787830342.502481:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830342.502486:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830342.502495:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00010000:00010000:2.0:1787830342.502502:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830342.502509:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671280999808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830342.502571:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830342/real 1787830342] req@ffff8ab3d0a412c0 x1874671280999680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830342.502582:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671280999680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830342.502595:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830342.502599:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830342.502603:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671280999680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830342.502648:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280999808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830342.502672:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830342.502685:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830342/real 1787830342] req@ffff8ab3d0a43840 x1874671280999808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830342.502697:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671280999808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830342.502705:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830342.502708:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830342.502711:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280999808:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787830342.502758:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830342.502761:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830342.502763:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830342.502767:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4892 00000100:00080000:3.0:1787830342.502771:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830342.502775:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830342.502781:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830342.502794:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671280999936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830342.502804:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830342.502806:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830342.502808:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830342.502812:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4892 00000100:00080000:3.0:1787830342.502815:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830342.502819:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830342.502824:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830342.502834:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281000064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830342.502843:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830342.502846:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830342.502847:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830342.502850:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4892 00000100:00080000:3.0:1787830342.502853:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830342.502857:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830342.502861:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830342.502870:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281000192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830342.502975:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280999936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830342.502997:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830342.503009:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830342/real 1787830342] req@ffff8ab3d0a43480 x1874671280999936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830342.503020:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671280999936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830342.503033:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830342.503035:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830342.503038:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671280999936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830342.503044:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281000064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830342.503063:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830342.503072:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830342/real 1787830342] req@ffff8ab3d0a43840 x1874671281000064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830342.503082:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281000064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830342.503091:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830342.503094:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830342.503097:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281000064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830342.503112:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281000192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830342.503129:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830342.503140:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830342/real 1787830342] req@ffff8ab3d0a412c0 x1874671281000192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830342.503151:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281000192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830342.503160:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830342.503163:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830342.503166:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281000192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830343.526843:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830344.550193:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830345.574328:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830346.598388:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830347.622053:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830347.622063:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830347.622067:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830347.622077:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4897 00000100:00080000:3.0:1787830347.622104:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830347.622112:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830347.622148:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281000320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787830347.622162:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830347.622184:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830347.622188:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830347.622190:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830347.622193:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4897 00000100:00080000:3.0:1787830347.622197:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830347.622202:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830347.622209:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830347.622231:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281000448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830347.622237:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830347.622239:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830347.622252:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830347.622258:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4897 00000100:00080000:3.0:1787830347.622262:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830347.622267:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830347.622272:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830347.622279:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281000320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830347.622291:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281000576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830347.622302:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830347.622305:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830347.622307:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830347.622310:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4897 00000100:00080000:3.0:1787830347.622313:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830347.622318:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830347.622322:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830347.622334:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281000704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830347.622347:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830347.622350:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830347.622351:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830347.622355:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4897 00000100:00080000:3.0:1787830347.622357:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830347.622361:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830347.622365:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830347.622375:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281000832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830347.622470:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281000448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830347.622508:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281000576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830347.622530:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281000704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830347.622559:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281000832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830348.647468:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830349.670031:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787830350.694306:0:10944:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:0.0:1787830350.694349:0:10944:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787830350.694365:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787830350.694449:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787830350.694458:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787830350.694466:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:0.0:1787830350.694476:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:0.0:1787830350.694542:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830347/real 1787830350] req@ffff8ab3d0a43c00 x1874671281000576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830350.694565:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281000576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830350.694577:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830350.694582:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830350.694586:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281000576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830350.694594:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830347/real 1787830350] req@ffff8ab3d0a40000 x1874671281000704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830350.694604:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281000704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830350.694613:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830350.694617:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830350.694620:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281000704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830350.694625:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830347/real 1787830350] req@ffff8ab3d0a421c0 x1874671281000832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830350.694644:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281000832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830350.694653:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830350.694656:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830350.694659:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281000832:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830350.694665:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830347/real 1787830350] req@ffff8ab3d0a430c0 x1874671281000320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830350.694676:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281000320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830350.694683:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830350.694685:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830350.694688:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281000320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830350.694693:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830347/real 1787830350] req@ffff8ab3d0a41e00 x1874671281000448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830350.694703:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281000448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830350.694710:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830350.694713:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830350.694716:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281000448:192.168.203.124@tcp:38:kworker.0 00010000:00010000:2.0:1787830350.694855:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830351.718109:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830352.742059:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830352.742067:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830352.742071:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830352.742079:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4902 00000100:00080000:3.0:1787830352.742088:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830352.742094:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830352.742126:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281000960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830352.742150:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830352.742153:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830352.742155:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830352.742159:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4902 00000100:00080000:3.0:1787830352.742162:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830352.742167:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830352.742176:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281000960:192.168.203.124@tcp:250:kworker.0 00000008:00100000:3.0:1787830352.742178:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830352.742195:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281001088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830352.742219:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830352.742223:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830352.742224:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830352.742228:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4902 00000100:00080000:3.0:1787830352.742244:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830352.742249:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830352.742254:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830352.742264:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281001216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830352.742271:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830352.742275:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830352.742276:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830352.742280:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4902 00000100:00080000:3.0:1787830352.742282:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830352.742286:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830352.742289:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830352.742297:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281001344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830352.742306:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787830352.742308:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830352.742310:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830352.742313:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830352.742318:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4902 00000100:00080000:3.0:1787830352.742322:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830352.742329:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830352.742333:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830352.742343:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281001472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787830352.742602:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830352.742960:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830352/real 1787830352] req@ffff8ab3d0a41e00 x1874671281000960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830352.742984:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281000960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830352.742993:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830352.742998:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830352.743002:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281000960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830352.743026:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281001088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830352.743058:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830352.743074:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830352/real 1787830352] req@ffff8ab3d0a430c0 x1874671281001088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830352.743092:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281001088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830352.743100:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830352.743103:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830352.743106:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281001088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830352.743112:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281001216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830352.743131:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830352.743142:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830352/real 1787830352] req@ffff8ab3d0a421c0 x1874671281001216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830352.743152:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281001216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830352.743159:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830352.743163:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830352.743165:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281001216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830352.743171:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281001344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830352.743189:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830352.743199:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830352/real 1787830352] req@ffff8ab3d0a40000 x1874671281001344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830352.743223:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281001344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830352.743231:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830352.743234:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830352.743236:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281001344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830352.743242:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281001472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830352.743259:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830352.743269:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830352/real 1787830352] req@ffff8ab3d0a43c00 x1874671281001472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830352.743294:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281001472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830352.743302:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830352.743305:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830352.743308:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281001472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830353.766237:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830354.790644:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830355.814098:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830356.838256:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830357.861956:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830357.861966:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830357.861971:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830357.861981:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4907 00000100:00080000:3.0:1787830357.861991:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830357.861999:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787830357.862033:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830357.862035:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281001600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830357.862064:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830357.862067:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830357.862069:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830357.862073:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4907 00000100:00080000:3.0:1787830357.862076:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830357.862079:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830357.862089:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830357.862103:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281001728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830357.862105:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281001600:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787830357.862127:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830357.862131:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830357.862132:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830357.862137:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4907 00000100:00080000:3.0:1787830357.862141:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830357.862145:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830357.862150:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830357.862162:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281001856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830357.862186:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830357.862191:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830357.862193:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830357.862197:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4907 00000100:00080000:3.0:1787830357.862201:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830357.862205:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830357.862210:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830357.862222:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281001984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830357.862232:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830357.862232:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830357.862238:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830357.862240:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830357.862245:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4907 00000100:00080000:3.0:1787830357.862250:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830357.862255:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830357.862259:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830357.862271:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281002112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830357.862284:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830357/real 1787830357] req@ffff8ab3d0a412c0 x1874671281001600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830357.862302:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281001600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830357.862312:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830357.862316:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830357.862320:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281001600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830357.862342:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281001728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830357.862365:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830357.862386:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830357/real 1787830357] req@ffff8ab3d0a43840 x1874671281001728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830357.862398:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281001728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830357.862411:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830357.862414:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830357.862418:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281001728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830357.862424:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281001856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830357.862448:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830357.862459:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830357/real 1787830357] req@ffff8ab3d0a43480 x1874671281001856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830357.862470:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281001856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830357.862477:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830357.862480:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830357.862483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281001856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830357.862489:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281001984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830357.862506:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830357.862515:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830357/real 1787830357] req@ffff8ab3d0a43c00 x1874671281001984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830357.862525:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281001984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830357.862533:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830357.862536:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830357.862539:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281001984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830357.862544:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281002112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830357.862557:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830357.862566:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830357/real 1787830357] req@ffff8ab3d0a40000 x1874671281002112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830357.862575:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281002112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830357.862584:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830357.862586:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830357.862589:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281002112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830358.886376:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830359.910238:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830360.935491:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830361.958881:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830362.982032:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830362.982042:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830362.982046:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830362.982056:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4912 00000100:00080000:0.0:1787830362.982066:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830362.982073:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787830362.982087:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830362.982110:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671281002240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830362.982133:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830362.982149:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830362.982151:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830362.982156:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4912 00000100:00080000:0.0:1787830362.982160:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830362.982165:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830362.982171:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281002240:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787830362.982187:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830362.982205:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671281002368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830362.982223:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830362.982227:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830362.982229:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787830362.982233:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830362.982235:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4912 00000100:00080000:0.0:1787830362.982241:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830362.982246:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830362.982251:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830362.982263:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671281002496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830362.982273:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830362.982276:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830362.982278:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830362.982284:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4912 00000100:00100000:2.0:1787830362.982284:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830362/real 1787830362] req@ffff8ab3f075da40 x1874671281002240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830362.982300:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830362.982308:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830362.982308:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671281002240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787830362.982315:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:2.0:1787830362.982323:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830362.982330:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830362.982331:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671281002624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830362.982335:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281002240:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830362.982350:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830362.982354:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830362.982356:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830362.982361:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4912 00000100:00100000:2.0:1787830362.982364:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281002368:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787830362.982367:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830362.982375:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830362.982380:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830362.982392:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671281002752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830362.982395:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830362.982412:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830362/real 1787830362] req@ffff8ab3f075ed00 x1874671281002368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830362.982425:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671281002368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830362.982434:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830362.982437:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830362.982440:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281002368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830362.982446:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281002496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830362.982482:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830362.982493:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830362/real 1787830362] req@ffff8ab3f075f480 x1874671281002496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830362.982504:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671281002496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830362.982512:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830362.982515:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830362.982518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281002496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830362.982523:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281002624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830362.982541:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830362.982552:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830362/real 1787830362] req@ffff8ab3f075e580 x1874671281002624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830362.982569:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671281002624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830362.982578:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830362.982581:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830362.982584:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281002624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830362.982599:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281002752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830362.982616:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830362.982639:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830362/real 1787830362] req@ffff8ab3f075f840 x1874671281002752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830362.982650:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671281002752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830362.982659:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830362.982662:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830362.982665:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281002752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830364.006159:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830365.030230:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830366.054284:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830367.078452:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830368.102398:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830368.102406:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830368.102410:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830368.102420:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4917 00000100:00080000:2.0:1787830368.102430:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830368.102437:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830368.102470:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671281002880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830368.102501:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830368.102504:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830368.102506:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830368.102510:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4917 00000100:00080000:2.0:1787830368.102513:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830368.102517:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830368.102526:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830368.102539:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671281003008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830368.102548:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830368.102551:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830368.102552:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830368.102556:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4917 00000100:00080000:2.0:1787830368.102559:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830368.102563:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830368.102568:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830368.102578:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671281003136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830368.102587:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830368.102590:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830368.102591:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830368.102595:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4917 00000100:00080000:2.0:1787830368.102598:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830368.102602:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830368.102607:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830368.102618:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671281003264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830368.102627:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830368.102629:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830368.102631:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830368.102635:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4917 00000100:00080000:2.0:1787830368.102638:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830368.102642:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830368.102646:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830368.102657:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671281003392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830368.102733:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281002880:192.168.203.124@tcp:250:kworker.0 00000800:00000100:0.0:1787830368.102792:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830368.102834:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830368/real 1787830368] req@ffff8ab3f0438780 x1874671281002880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830368.102850:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671281002880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830368.102860:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830368.102864:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830368.102867:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281002880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830368.102875:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281003008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830368.102930:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830368.102949:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830368/real 1787830368] req@ffff8ab3f0438b40 x1874671281003008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830368.102963:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671281003008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830368.102978:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830368.102982:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00010000:00010000:1.0:1787830368.102985:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830368.102987:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281003008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830368.102997:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281003136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830368.103019:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830368.103031:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830368/real 1787830368] req@ffff8ab3f0438f00 x1874671281003136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830368.103043:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671281003136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830368.103052:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830368.103056:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830368.103060:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281003136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830368.103068:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281003264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830368.103091:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830368.103112:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830368/real 1787830368] req@ffff8ab3f043a580 x1874671281003264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830368.103123:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671281003264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830368.103132:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830368.103134:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830368.103137:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281003264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830368.103142:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281003392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830368.103160:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830368.103170:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830368/real 1787830368] req@ffff8ab3f043b0c0 x1874671281003392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830368.103180:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671281003392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830368.103189:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830368.103192:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830368.103195:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281003392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830369.126282:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830370.150117:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830371.174177:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830372.198051:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830373.222117:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830373.222128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830373.222132:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830373.222143:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4922 00000100:00080000:0.0:1787830373.222153:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830373.222162:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830373.222200:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671281003520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787830373.222225:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830373.222235:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830373.222240:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830373.222242:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830373.222246:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4922 00000100:00080000:0.0:1787830373.222250:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830373.222255:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830373.222265:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830373.222278:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671281003648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830373.222288:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830373.222294:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787830373.222294:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281003520:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830373.222297:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830373.222306:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4922 00000100:00080000:0.0:1787830373.222309:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830373.222313:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830373.222318:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830373.222331:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671281003776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830373.222341:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830373.222344:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830373.222347:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830373.222351:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4922 00000800:00000100:1.0:1787830373.222372:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830373.223183:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830373.223192:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830373.223200:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830373.223217:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671281003904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830373.223227:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830373/real 1787830373] req@ffff8ab3f075f0c0 x1874671281003520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830373.223228:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830373.223238:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830373.223242:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830373.223249:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4922 00000100:00100000:1.0:1787830373.223251:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671281003520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830373.223255:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830373.223262:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830373.223263:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787830373.223269:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830373.223272:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281003520:192.168.203.124@tcp:250:kworker.0 00000008:00100000:0.0:1787830373.223273:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830373.223289:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671281004032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830373.223295:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281003648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830373.223324:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830373.223335:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830373/real 1787830373] req@ffff8ab3f075cb40 x1874671281003648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830373.223344:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671281003648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830373.223350:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830373.223352:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830373.223354:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281003648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830373.223357:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281003776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830373.223367:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830373.223373:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830373/real 1787830373] req@ffff8ab3f075c000 x1874671281003776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830373.223379:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671281003776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830373.223384:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830373.223385:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830373.223387:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281003776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830373.223390:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281003904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830373.223399:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830373.223404:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830373/real 1787830373] req@ffff8ab3f075de00 x1874671281003904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830373.223410:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671281003904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830373.223419:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830373.223421:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830373.223423:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281003904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830373.223431:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281004032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830373.223442:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830373.223451:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830373/real 1787830373] req@ffff8ab3f075e940 x1874671281004032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830373.223461:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671281004032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830373.223469:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830373.223472:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830373.223475:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281004032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830374.246127:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830375.270185:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830376.294090:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830377.318085:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830378.342083:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830378.342269:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830378.342274:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830378.342278:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830378.342286:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4927 00000100:00080000:3.0:1787830378.342295:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830378.342299:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830378.342328:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281004160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830378.342346:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830378.342349:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830378.342350:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830378.342353:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4927 00000100:00080000:3.0:1787830378.342355:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830378.342357:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830378.342364:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830378.342373:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281004288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830378.342382:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830378.342387:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281004160:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787830378.342390:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830378.342395:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830378.342400:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4927 00000100:00080000:3.0:1787830378.342403:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830378.342408:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830378.342413:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830378.342425:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281004416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830378.342435:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830378.342438:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830378.342440:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787830378.342443:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830378.342449:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4927 00000100:00080000:3.0:1787830378.342453:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830378.342457:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830378.342462:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830378.342472:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281004544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830378.342477:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830378.342480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830378.342481:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830378.342483:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4927 00000100:00080000:3.0:1787830378.342486:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830378.342488:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830378.342492:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830378.342499:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281004672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830378.342525:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830378/real 1787830378] req@ffff8ab3d0a421c0 x1874671281004160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830378.342537:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281004160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830378.342558:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830378.342561:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830378.342564:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281004160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830378.342580:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281004288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830378.342594:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830378.342603:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830378/real 1787830378] req@ffff8ab3d0a430c0 x1874671281004288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830378.342610:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281004288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830378.342615:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830378.342617:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830378.342619:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281004288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830378.342622:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281004416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830378.342634:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830378.342806:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830378/real 1787830378] req@ffff8ab3d0a41e00 x1874671281004416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830378.342818:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281004416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830378.342826:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830378.342830:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830378.342833:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281004416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830378.342840:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281004544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830378.342857:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830378.342865:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830378/real 1787830378] req@ffff8ab3d0a40000 x1874671281004544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830378.342872:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281004544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830378.342878:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830378.342880:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830378.342882:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281004544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830378.342962:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281004672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830378.342984:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830378.342996:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830378/real 1787830378] req@ffff8ab3d0a43c00 x1874671281004672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830378.343007:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281004672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830378.343017:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830378.343020:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830378.343023:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281004672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830379.366546:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830380.390203:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830381.414091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830382.438053:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830383.462061:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830383.462072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830383.462077:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830383.462089:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4933 00000100:00080000:2.0:1787830383.462099:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830383.462107:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830383.462150:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671281004800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787830383.462231:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830383.462236:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281004800:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830383.462303:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830383.462338:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830383.462342:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830383.462344:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830383.462349:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4933 00000100:00080000:2.0:1787830383.462353:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830383.462357:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830383.462370:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830383.462385:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671281004928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830383.462577:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830383/real 1787830383] req@ffff8ab3f0439680 x1874671281004800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830383.462595:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671281004800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830383.462605:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830383.462610:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830383.462614:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281004800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830383.462635:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281004928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830383.462657:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830383.462670:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830383/real 1787830383] req@ffff8ab3f04392c0 x1874671281004928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830383.462681:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671281004928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830383.462689:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830383.462691:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830383.462695:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281004928:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787830383.462745:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830383.462749:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830383.462751:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830383.462756:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4933 00000100:00080000:2.0:1787830383.462760:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830383.462766:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830383.462771:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830383.462785:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671281005056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830383.462808:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830383.462812:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830383.462813:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830383.462817:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4933 00000100:00080000:2.0:1787830383.462821:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830383.462824:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830383.462829:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830383.462840:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671281005184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830383.462848:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830383.462852:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830383.462854:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830383.462857:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4933 00000100:00080000:2.0:1787830383.462860:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830383.462864:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830383.462868:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830383.462876:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671281005312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830383.462910:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281005056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830383.462942:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830383.462956:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830383/real 1787830383] req@ffff8ab3f043ad00 x1874671281005056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830383.462970:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671281005056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830383.462979:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830383.462983:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830383.462986:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281005056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830383.463006:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281005184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830383.463029:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830383.463041:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830383/real 1787830383] req@ffff8ab3f04383c0 x1874671281005184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830383.463053:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671281005184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830383.463063:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830383.463067:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830383.463070:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281005184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830383.463076:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281005312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830383.463094:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830383.463105:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830383/real 1787830383] req@ffff8ab3f043bc00 x1874671281005312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830383.463115:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671281005312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830383.463124:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830383.463127:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830383.463130:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281005312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830384.486236:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830385.510213:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830386.534148:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830387.558259:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830388.582031:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830388.582041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830388.582045:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830388.582057:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4938 00000100:00080000:2.0:1787830388.582068:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830388.582076:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:1.0:1787830388.582123:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830388.582265:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671281005440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830388.582296:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830388.582300:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830388.582301:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830388.582306:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4938 00000100:00080000:2.0:1787830388.582309:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830388.582314:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830388.582324:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830388.582336:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671281005568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830388.582345:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830388.582349:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830388.582351:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830388.582356:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4938 00000100:00080000:2.0:1787830388.582361:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830388.582365:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830388.582371:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830388.582380:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281005440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830388.582389:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671281005696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830388.582402:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830388.582406:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830388.582408:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830388.582413:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4938 00000100:00080000:2.0:1787830388.582417:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830388.582422:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830388.582427:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830388.582439:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671281005824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830388.582449:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830388.582452:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830388.582454:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830388.582459:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4938 00000100:00080000:2.0:1787830388.582463:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830388.582467:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830388.582472:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000800:00000100:1.0:1787830388.582474:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830388.582535:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830388/real 1787830388] req@ffff8ab3f0439a40 x1874671281005440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830388.582558:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671281005952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830388.582563:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671281005440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830388.582582:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830388.582587:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830388.582591:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281005440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830388.582600:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281005568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830388.582624:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830388.582641:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830388/real 1787830388] req@ffff8ab3f043a940 x1874671281005568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830388.582664:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671281005568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830388.582673:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830388.582677:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830388.582680:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281005568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830388.582709:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281005696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830388.582728:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830388.582738:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830388/real 1787830388] req@ffff8ab3f043a1c0 x1874671281005696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830388.582749:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671281005696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830388.582757:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830388.582760:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830388.582763:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281005696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830388.582769:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281005824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830388.582786:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830388.582797:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830388/real 1787830388] req@ffff8ab3f0439e00 x1874671281005824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830388.582808:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671281005824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830388.582824:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830388.582827:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830388.582830:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281005824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830388.582836:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281005952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830388.582867:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830388.582877:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830388/real 1787830388] req@ffff8ab3f043b840 x1874671281005952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830388.582912:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671281005952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830388.582922:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830388.582925:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830388.582928:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281005952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830389.608100:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830390.630406:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830391.654235:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830392.678024:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830393.702216:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830393.702273:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830393.702289:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830393.702294:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830393.702304:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4943 00000100:00080000:3.0:1787830393.702315:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830393.702321:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830393.702357:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281006080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830393.702396:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830393.702400:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830393.702401:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830393.702405:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4943 00000100:00080000:3.0:1787830393.702408:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830393.702420:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830393.702430:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830393.702442:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281006208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830393.702451:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830393.702454:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830393.702455:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830393.702459:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4943 00000100:00080000:3.0:1787830393.702462:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830393.702466:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830393.702471:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830393.702481:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281006336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830393.702490:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830393.702493:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830393.702495:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830393.702498:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4943 00000100:00080000:3.0:1787830393.702506:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830393.702510:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830393.702515:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830393.702528:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281006464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830393.702539:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830393.702543:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281006080:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787830393.702543:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830393.702546:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830393.702555:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4943 00000100:00080000:3.0:1787830393.702558:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830393.702562:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830393.702566:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830393.702577:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281006592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830393.702608:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830393.702653:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830393/real 1787830393] req@ffff8ab3d0a43480 x1874671281006080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830393.702670:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281006080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830393.702680:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830393.702685:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830393.702688:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281006080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830393.702695:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281006208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830393.702726:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830393.702738:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830393/real 1787830393] req@ffff8ab3d0a43840 x1874671281006208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830393.702756:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281006208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830393.702764:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830393.702768:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830393.702771:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281006208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830393.702777:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281006336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830393.702796:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830393.702808:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830393/real 1787830393] req@ffff8ab3d0a412c0 x1874671281006336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830393.702819:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281006336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830393.702827:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830393.702830:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830393.702834:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281006336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830393.702856:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281006464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830393.702875:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830393.702905:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830393/real 1787830393] req@ffff8ab3d0a43c00 x1874671281006464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830393.702918:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281006464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830393.702936:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830393.702940:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830393.702943:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281006464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830393.702949:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281006592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830393.702966:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830393.702977:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830393/real 1787830393] req@ffff8ab3d0a40000 x1874671281006592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830393.702988:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281006592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830393.702996:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830393.702999:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830393.703006:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281006592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830394.726271:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830395.750138:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830396.774558:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830397.798151:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830398.822066:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830398.823021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830398.823028:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830398.823033:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830398.823042:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4948 00000100:00080000:2.0:1787830398.823053:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830398.823062:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830398.823099:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671281006720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830398.823267:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281006720:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830398.823316:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830398.823335:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830398.823340:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830398.823342:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830398.823348:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4948 00000100:00080000:2.0:1787830398.823353:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830398.823359:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830398.823363:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830398/real 1787830398] req@ffff8ab3f043b480 x1874671281006720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787830398.823371:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830398.823381:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671281006720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830398.823391:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671281006848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830398.823393:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830398.823402:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830398.823410:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281006720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787830398.823412:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830398.823418:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830398.823420:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830398.823427:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4948 00000100:00080000:2.0:1787830398.823431:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830398.823436:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830398.823442:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830398.823444:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281006848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830398.823458:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671281006976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830398.823468:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830398.823475:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830398.823480:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787830398.823483:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830398/real 1787830398] req@ffff8ab3f043b840 x1874671281006848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830398.823484:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830398.823495:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4948 00000100:00100000:1.0:1787830398.823498:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671281006848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830398.823501:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830398.823509:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830398.823512:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787830398.823514:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830398.823521:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281006848:192.168.203.124@tcp:38:kworker.0 00000008:00100000:2.0:1787830398.823524:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830398.823530:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281006976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830398.823540:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671281007104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830398.823546:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830398.823557:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830398/real 1787830398] req@ffff8ab3f0439e00 x1874671281006976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830398.823559:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830398.823569:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830398.823572:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830398.823575:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671281006976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830398.823579:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4948 00000100:00080000:2.0:1787830398.823586:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830398.823593:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787830398.823597:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787830398.823600:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787830398.823604:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830398.823610:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281006976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830398.823615:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671281007232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830398.823626:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281007104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830398.823650:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830398.823676:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830398/real 1787830398] req@ffff8ab3f043a1c0 x1874671281007104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830398.823686:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671281007104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830398.823695:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830398.823698:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830398.823701:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281007104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830398.823712:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281007232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830398.823730:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830398.823749:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830398/real 1787830398] req@ffff8ab3f043a940 x1874671281007232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830398.823760:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671281007232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830398.823769:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830398.823771:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830398.823775:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281007232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830399.847035:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830400.871040:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830401.894539:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830402.918658:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830403.942119:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830403.942198:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830403.942206:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830403.942211:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830403.942222:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4953 00000100:00080000:0.0:1787830403.942232:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830403.942240:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830403.942274:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671281007360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830403.942297:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830403.942301:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830403.942303:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830403.942307:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4953 00000100:00080000:0.0:1787830403.942312:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830403.942317:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830403.942326:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830403.942337:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281007360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830403.942344:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671281007488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830403.942356:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830403.942360:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830403.942362:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830403.942367:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4953 00000100:00080000:0.0:1787830403.942371:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830403.942377:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830403.942382:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830403.942394:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671281007616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830403.942403:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830403.942405:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830403.942410:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830403.942412:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830403.942417:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4953 00000100:00080000:0.0:1787830403.942420:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830403.942425:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830403.942429:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830403.942440:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671281007744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830403.942448:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830403.942452:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830403/real 1787830403] req@ffff8ab3f075cf00 x1874671281007360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830403.942453:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830403.942456:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830403.942464:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4953 00000100:00080000:0.0:1787830403.942469:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830403.942475:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830403.942480:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830403.942484:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671281007360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830403.942496:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830403.942501:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830403.942581:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281007360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830403.942619:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671281007872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830403.942716:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281007488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830403.942752:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830403.942772:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830403/real 1787830403] req@ffff8ab3f075e940 x1874671281007488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830403.942787:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671281007488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830403.942796:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830403.942801:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830403.942804:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281007488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830403.942811:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281007616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830403.942831:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830403.942843:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830403/real 1787830403] req@ffff8ab3f075de00 x1874671281007616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830403.943061:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671281007616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830403.943076:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830403.943079:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830403.943083:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281007616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830403.943090:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281007744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830403.943112:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830403.943125:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830403/real 1787830403] req@ffff8ab3f075c000 x1874671281007744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830403.943138:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671281007744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830403.943146:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830403.943150:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830403.943153:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281007744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830403.943159:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281007872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830403.943176:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830403.943194:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830403/real 1787830403] req@ffff8ab3f075cb40 x1874671281007872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830403.943206:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671281007872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830403.943215:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830403.943217:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830403.943221:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281007872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830404.966078:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830405.990326:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830407.014052:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830408.038688:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830409.062430:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830409.062439:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830409.062443:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830409.062453:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4958 00000100:00080000:3.0:1787830409.062463:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830409.062469:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830409.062502:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281008000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830409.062533:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830409.062536:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830409.062538:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830409.062542:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4958 00000100:00080000:3.0:1787830409.062546:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830409.062549:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830409.062558:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00010000:00010000:1.0:1787830409.062565:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830409.062571:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281008128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830409.062579:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281008000:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787830409.062583:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830409.062588:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830409.062590:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830409.062595:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4958 00000100:00080000:3.0:1787830409.062597:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830409.062601:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830409.062605:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830409.062616:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281008256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830409.062624:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830409.062626:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830409.062628:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830409.062631:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4958 00000100:00080000:3.0:1787830409.062633:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830409.062637:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830409.062640:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:0.0:1787830409.062643:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830409.062653:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281008384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830409.062662:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830409.062665:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830409.062666:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830409.062670:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4958 00000100:00080000:3.0:1787830409.062672:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830409.062675:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830409.062679:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830409.062688:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281008512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830409.062696:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830409/real 1787830409] req@ffff8ab3d0a41e00 x1874671281008000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830409.062714:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281008000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830409.062725:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830409.062729:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830409.062733:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281008000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830409.062754:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281008128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830409.062776:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830409.062787:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830409/real 1787830409] req@ffff8ab3d0a430c0 x1874671281008128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830409.062798:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281008128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830409.062806:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830409.062809:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830409.062813:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281008128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830409.062829:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281008256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830409.062847:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830409.062858:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830409/real 1787830409] req@ffff8ab3d0a421c0 x1874671281008256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830409.062868:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281008256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830409.062903:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830409.062907:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830409.062911:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281008256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830409.062917:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281008384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830409.062936:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830409.062947:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830409/real 1787830409] req@ffff8ab3d0a40000 x1874671281008384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830409.062956:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281008384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830409.062969:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830409.062971:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830409.062974:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281008384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830409.062979:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281008512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830409.062994:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830409.063003:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830409/real 1787830409] req@ffff8ab3d0a43c00 x1874671281008512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830409.063012:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281008512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830409.063021:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830409.063024:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830409.063027:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281008512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830410.088623:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830411.110969:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830412.134043:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830413.158075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830414.182072:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830414.182141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830414.182146:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830414.182150:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830414.182156:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4963 00000100:00080000:0.0:1787830414.182163:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830414.182168:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830414.182208:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671281008640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830414.182277:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281008640:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830414.182500:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830414.182504:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830414.182506:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830414.182509:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4963 00000100:00080000:0.0:1787830414.182512:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830414.182515:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830414.182522:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830414.182535:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671281008768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830414.182569:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281008768:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787830414.182629:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830414.182633:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830414.182635:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830414.182641:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4963 00000100:00080000:0.0:1787830414.182645:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830414.182649:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830414.182654:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830414.182666:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671281008896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830414.182676:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830414.182679:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830414.182681:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830414.182685:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4963 00000100:00080000:0.0:1787830414.182689:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830414.182693:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830414.182698:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830414.182708:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671281009024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830414.182716:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830414.182719:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830414.182721:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830414.182725:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4963 00000100:00080000:0.0:1787830414.182728:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830414.182731:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830414.182735:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830414.182743:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671281009152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830414.182822:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281008896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830414.182849:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281009024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830414.182871:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281009152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830415.206817:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830416.230146:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830417.254189:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787830417.254366:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:3.0:1787830417.254405:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787830417.254421:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:0.0:1787830417.255729:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830414/real 1787830417] req@ffff8ab3f075f0c0 x1874671281008640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830439 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830417.255835:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671281008640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830439 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830417.255850:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830417.255855:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830417.255859:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281008640:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830417.257870:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830417.261207:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830417.261223:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830417.261230:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:0.0:1787830417.261415:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830414/real 1787830417] req@ffff8ab3f075e580 x1874671281008896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830417.261438:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671281008896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830417.261449:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830417.261454:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830417.261458:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281008896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830417.261465:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830414/real 1787830417] req@ffff8ab3f075f480 x1874671281009024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830417.261477:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671281009024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830417.261500:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830417.261504:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830417.261508:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281009024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830417.261514:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830414/real 1787830417] req@ffff8ab3f075ed00 x1874671281009152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830417.261526:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671281009152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830417.261535:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830417.261538:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830417.261542:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281009152:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830417.261549:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830414/real 1787830417] req@ffff8ab3f075f840 x1874671281008768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830417.261561:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671281008768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830417.261567:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830417.261571:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830417.261573:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281008768:192.168.203.124@tcp:38:kworker.0 00010000:00010000:3.0:1787830418.278132:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830419.302004:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830419.302012:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830419.302016:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830419.302023:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4968 00000100:00080000:2.0:1787830419.302032:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830419.302038:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787830419.302047:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830419.302066:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671281009280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830419.302085:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830419.302088:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830419.302090:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830419.302094:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4968 00000100:00080000:2.0:1787830419.302097:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830419.302102:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830419.302111:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830419.302118:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281009280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830419.302127:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671281009408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830419.302137:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830419.302140:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830419.302142:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830419.302146:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4968 00000100:00080000:2.0:1787830419.302162:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830419.302166:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830419.302170:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787830419.302180:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830419.302182:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671281009536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830419.302198:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830419.302201:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830419.302203:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830419.302208:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4968 00000100:00080000:2.0:1787830419.302212:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830419.302217:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830419.302220:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830419/real 1787830419] req@ffff8ab3f0439a40 x1874671281009280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787830419.302223:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830419.302243:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671281009280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830419.302243:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671281009664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830419.302273:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830419.302274:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830419.302280:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00080000:2.0:1787830419.302281:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787830419.302285:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281009280:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787830419.302285:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830419.302293:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4968 00000100:00080000:2.0:1787830419.302297:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830419.302302:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830419.302306:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830419.302313:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281009408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830419.302319:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671281009792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830419.302342:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830419.302356:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830419/real 1787830419] req@ffff8ab3f043bc00 x1874671281009408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830419.302378:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671281009408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830419.302383:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830419.302385:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830419.302387:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281009408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830419.302391:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281009536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830419.302408:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830419.302418:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830419/real 1787830419] req@ffff8ab3f04383c0 x1874671281009536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830419.302428:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671281009536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830419.302436:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830419.302439:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830419.302444:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281009536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830419.302451:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281009664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830419.302468:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830419.302478:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830419/real 1787830419] req@ffff8ab3f043ad00 x1874671281009664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830419.302488:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671281009664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830419.302502:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830419.302506:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830419.302508:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281009664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830419.302523:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281009792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830419.302541:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830419.302552:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830419/real 1787830419] req@ffff8ab3f04392c0 x1874671281009792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830419.302563:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671281009792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830419.302572:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830419.302575:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830419.302578:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281009792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830420.326302:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830421.350308:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830422.374067:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830423.398443:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830424.422064:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830424.422071:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830424.422074:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830424.422082:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4974 00000100:00080000:2.0:1787830424.422092:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830424.422098:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830424.422129:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671281009920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830424.422207:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281009920:192.168.203.124@tcp:250:kworker.0 00000800:00000100:0.0:1787830424.422267:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830424.422297:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830424.422301:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830424.422303:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830424.422307:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4974 00000100:00080000:2.0:1787830424.422311:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830424.422315:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830424.422329:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830424.422344:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671281010048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830424.422407:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830424/real 1787830424] req@ffff8ab3f0439680 x1874671281009920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830424.422433:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671281009920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830424.422635:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830424.422640:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830424.422644:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281009920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830424.422666:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281010048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830424.422691:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830424.422704:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830424/real 1787830424] req@ffff8ab3f043b0c0 x1874671281010048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830424.422729:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671281010048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830424.422737:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830424.422740:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830424.422743:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281010048:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787830424.422846:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830424.422850:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830424.422852:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830424.422856:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4974 00000100:00080000:2.0:1787830424.422860:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830424.422864:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830424.422870:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830424.422883:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671281010176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830424.422946:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281010176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830424.422966:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830424.422985:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830424.422989:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830424.422991:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830424.422996:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4974 00000100:00080000:2.0:1787830424.423000:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830424.423004:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830424.423009:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830424.423020:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671281010304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830424.423058:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830424/real 1787830424] req@ffff8ab3f043a580 x1874671281010176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830424.423069:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671281010176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830424.423076:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830424.423079:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830424.423082:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281010176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830424.423094:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281010304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830424.423114:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830424.423125:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830424/real 1787830424] req@ffff8ab3f0438f00 x1874671281010304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830424.423137:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671281010304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830424.423147:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830424.423150:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830424.423154:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281010304:192.168.203.124@tcp:8:kworker.0 00000100:00080000:2.0:1787830424.423195:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830424.423199:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830424.423201:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830424.423206:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4974 00000100:00080000:2.0:1787830424.423210:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830424.423214:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830424.423219:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830424.423232:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671281010432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830424.423294:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281010432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830424.423315:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830424.423326:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830424/real 1787830424] req@ffff8ab3f0438b40 x1874671281010432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830424.423338:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671281010432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830424.423348:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830424.423351:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830424.423354:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281010432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830424.424046:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830425.447083:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830426.470114:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830427.494801:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830428.518129:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830429.542500:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830429.542584:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830429.542593:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830429.542598:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830429.542609:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4979 00000100:00080000:3.0:1787830429.542621:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830429.542630:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830429.542690:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281010560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830429.542715:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830429.542718:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830429.542719:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830429.542721:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4979 00000100:00080000:3.0:1787830429.542724:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830429.542727:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830429.542739:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830429.542754:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281010688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830429.542766:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830429.542775:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281010560:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787830429.542775:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830429.542778:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830429.542791:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4979 00000100:00080000:3.0:1787830429.542795:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830429.542799:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830429.542804:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830429.542818:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281010816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830429.542826:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830429.542830:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830429.542833:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830429.542837:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4979 00000100:00080000:3.0:1787830429.542849:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830429.542857:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:0.0:1787830429.542858:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787830429.542868:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830429.542924:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281010944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830429.542929:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830429/real 1787830429] req@ffff8ab3d0a412c0 x1874671281010560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830429.542964:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830429.542980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830429.542987:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830429.542998:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4979 00000100:00080000:3.0:1787830429.543004:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830429.543008:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281010560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830429.543009:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830429.543015:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:0.0:1787830429.543030:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787830429.543030:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281011072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830429.543046:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830429.543061:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281010560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830429.543088:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281010688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830429.543584:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830429.543601:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830429/real 1787830429] req@ffff8ab3d0a43840 x1874671281010688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830429.543628:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281010688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830429.543637:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830429.543640:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830429.543643:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281010688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830429.543650:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281010816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830429.543682:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830429.543693:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830429/real 1787830429] req@ffff8ab3d0a43480 x1874671281010816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830429.543705:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281010816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830429.543718:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830429.543722:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830429.543725:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281010816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830429.543731:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281010944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830429.543751:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830429.543762:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830429/real 1787830429] req@ffff8ab3d0a43c00 x1874671281010944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830429.543786:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281010944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830429.543796:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830429.543799:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830429.543802:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281010944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830429.543809:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281011072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830429.543827:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830429.543837:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830429/real 1787830429] req@ffff8ab3d0a40000 x1874671281011072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830429.543848:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281011072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830429.543856:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830429.543859:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830429.543863:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281011072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830430.566122:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830431.590370:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830432.615752:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830433.638171:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830434.662391:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830434.662418:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830434.662436:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830434.662442:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830434.662454:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4984 00000100:00080000:2.0:1787830434.662466:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830434.662474:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830434.662523:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671281011200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830434.662554:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830434.662558:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830434.662560:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830434.662564:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4984 00000100:00080000:2.0:1787830434.662567:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830434.662571:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830434.662583:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830434.662597:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671281011328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830434.662606:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830434.662609:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830434.662611:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830434.662615:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4984 00000100:00080000:2.0:1787830434.662618:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830434.662622:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830434.662626:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830434.662648:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671281011456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830434.662663:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830434.662666:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830434.662668:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830434.662672:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4984 00000100:00080000:2.0:1787830434.662675:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830434.662679:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830434.662684:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830434.662695:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671281011584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830434.662703:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830434.662707:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830434.662708:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830434.662712:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4984 00000100:00080000:2.0:1787830434.662715:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830434.662719:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830434.662723:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830434.662731:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671281011712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830434.662932:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281011200:192.168.203.124@tcp:250:kworker.0 00000800:00000100:0.0:1787830434.663007:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830434.663065:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830434/real 1787830434] req@ffff8ab3f0438780 x1874671281011200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830434.663084:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671281011200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830434.663095:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830434.663100:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830434.663104:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281011200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830434.663112:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281011328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830434.663134:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830434.663147:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830434/real 1787830434] req@ffff8ab3f0438b40 x1874671281011328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830434.663159:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671281011328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830434.663167:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830434.663170:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830434.663173:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281011328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830434.663179:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281011456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830434.663196:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830434.663207:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830434/real 1787830434] req@ffff8ab3f0438f00 x1874671281011456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830434.663218:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671281011456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830434.663225:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830434.663228:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830434.663231:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281011456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830434.663237:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281011584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830434.663253:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830434.663263:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830434/real 1787830434] req@ffff8ab3f043a580 x1874671281011584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830434.663274:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671281011584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830434.663283:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830434.663287:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830434.663290:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281011584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830434.663296:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281011712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830434.663314:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830434.663325:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830434/real 1787830434] req@ffff8ab3f043b0c0 x1874671281011712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830434.663336:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671281011712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830434.663347:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830434.663351:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830434.663354:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281011712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830435.686101:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830436.710731:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830437.735996:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830438.761709:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830439.782011:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830439.782021:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830439.782026:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830439.782037:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4989 00000100:00080000:2.0:1787830439.782047:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830439.782054:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830439.782099:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671281011840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830439.782123:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830439.782127:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830439.782128:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830439.782139:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4989 00000100:00080000:2.0:1787830439.782144:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830439.782148:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830439.782160:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830439.782186:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671281011968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830439.782195:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281011840:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787830439.782201:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830439.782206:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830439.782208:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830439.782214:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4989 00000100:00080000:2.0:1787830439.782219:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830439.782224:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830439.782241:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830439.782257:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671281012096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830439.782283:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830439.782286:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830439.782288:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830439.782292:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4989 00000100:00080000:2.0:1787830439.782296:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830439.782301:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830439.782315:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00010000:00010000:3.0:1787830439.782316:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830439.782329:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671281012224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830439.782339:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830439.782342:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830439.782343:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830439.782347:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4989 00000100:00080000:2.0:1787830439.782351:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830439.782355:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830439.782359:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830439.782368:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671281012352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830439.782458:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830439.782580:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830439/real 1787830439] req@ffff8ab3f0439680 x1874671281011840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830439.782611:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671281011840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830439.782622:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830439.782626:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830439.782630:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281011840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830439.782654:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281011968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830439.782680:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830439.782696:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830439/real 1787830439] req@ffff8ab3f04392c0 x1874671281011968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830439.782708:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671281011968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830439.782716:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830439.782720:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830439.782724:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281011968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830439.782732:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281012096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830439.782858:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830439.782871:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830439/real 1787830439] req@ffff8ab3f043ad00 x1874671281012096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830439.782884:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671281012096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830439.782924:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830439.782928:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830439.782932:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281012096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830439.783018:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281012224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830439.783041:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830439.783053:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830439/real 1787830439] req@ffff8ab3f04383c0 x1874671281012224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830439.783063:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671281012224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830439.783069:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830439.783071:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830439.783073:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281012224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830439.783077:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281012352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830439.783090:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830439.783096:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830439/real 1787830439] req@ffff8ab3f043bc00 x1874671281012352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830439.783103:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671281012352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830439.783109:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830439.783110:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830439.783112:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281012352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830440.807086:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830441.830150:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830442.854523:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830443.878157:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830444.902107:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830444.902170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830444.902176:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830444.902180:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830444.902190:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4994 00000100:00080000:3.0:1787830444.902201:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830444.902207:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830444.902244:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281012480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830444.902271:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830444.902275:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830444.902276:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830444.902280:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4994 00000100:00080000:3.0:1787830444.902284:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830444.902288:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830444.902297:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830444.902309:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281012608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830444.902320:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830444.902322:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281012480:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787830444.902324:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830444.902328:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830444.902334:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4994 00000100:00080000:3.0:1787830444.902337:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830444.902341:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830444.902347:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830444.902358:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281012736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830444.902367:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830444.902371:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830444.902373:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830444.902387:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4994 00000800:00000100:0.0:1787830444.902389:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830444.902392:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830444.902398:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830444.902404:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830444.902417:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281012864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830444.902426:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830444.902429:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830444.902431:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830444.902435:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4994 00000100:00080000:3.0:1787830444.902438:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830444.902443:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830444.902448:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830444.902450:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830444/real 1787830444] req@ffff8ab3d0a421c0 x1874671281012480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830444.902460:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281012992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830444.902474:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281012480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830444.902486:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830444.902495:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830444.902499:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281012480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830444.902522:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281012608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830444.902550:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830444.902563:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830444/real 1787830444] req@ffff8ab3d0a430c0 x1874671281012608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830444.902588:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281012608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830444.902602:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830444.902606:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830444.902609:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281012608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830444.902614:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281012736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830444.902629:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830444.902636:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830444/real 1787830444] req@ffff8ab3d0a41e00 x1874671281012736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830444.902643:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281012736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830444.902648:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830444.902650:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830444.902652:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281012736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830444.902655:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281012864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830444.902668:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830444.902677:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830444/real 1787830444] req@ffff8ab3d0a40000 x1874671281012864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830444.902686:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281012864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830444.902696:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830444.902699:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830444.902703:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281012864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830444.902708:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281012992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830444.902723:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830444.902730:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830444/real 1787830444] req@ffff8ab3d0a43c00 x1874671281012992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830444.902737:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281012992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830444.902743:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830444.902745:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830444.902747:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281012992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830445.926048:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830446.950443:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830447.974047:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830449.000921:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830450.022163:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830450.022173:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830450.022177:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830450.022188:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 4999 00000100:00080000:3.0:1787830450.022198:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830450.022206:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830450.022245:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281013120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830450.022279:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830450.022283:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830450.022284:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830450.022288:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4999 00000100:00080000:3.0:1787830450.022292:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830450.022296:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830450.022305:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830450.022316:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281013248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830450.022325:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830450.022328:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830450.022330:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830450.022333:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4999 00000100:00080000:3.0:1787830450.022337:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830450.022341:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830450.022345:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830450.022355:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281013376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830450.022364:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830450.022366:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830450.022368:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830450.022372:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4999 00000100:00080000:3.0:1787830450.022375:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830450.022379:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830450.022383:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830450.022402:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281013504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830450.022411:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830450.022414:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830450.022416:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830450.022419:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 4999 00000100:00080000:3.0:1787830450.022423:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830450.022426:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830450.022431:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830450.022440:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281013632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830450.022477:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281013120:192.168.203.124@tcp:250:kworker.0 00000800:00000100:0.0:1787830450.022547:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00010000:00010000:3.0:1787830450.022639:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830450.022652:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830450/real 1787830450] req@ffff8ab3d0a43480 x1874671281013120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830450.022671:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281013120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830450.022681:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830450.022685:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830450.022689:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281013120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830450.022696:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281013248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830450.022718:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830450.022731:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830450/real 1787830450] req@ffff8ab3d0a43840 x1874671281013248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830450.022743:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281013248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830450.022752:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830450.022755:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830450.022759:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281013248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830450.022764:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281013376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830450.022783:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830450.022877:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830450/real 1787830450] req@ffff8ab3d0a412c0 x1874671281013376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830450.022922:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281013376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830450.022931:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830450.022934:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830450.022937:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281013376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830450.022944:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281013504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830450.022964:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830450.022975:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830450/real 1787830450] req@ffff8ab3d0a43c00 x1874671281013504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830450.022986:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281013504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830450.022996:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830450.022999:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830450.023003:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281013504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830450.023009:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281013632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830450.023023:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830450.023030:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830450/real 1787830450] req@ffff8ab3d0a40000 x1874671281013632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830450.023037:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281013632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830450.023043:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830450.023045:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830450.023047:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281013632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830451.046086:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830452.070693:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830453.094561:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830454.118203:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830455.142011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830455.142026:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830455.142031:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830455.142044:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5004 00000100:00080000:3.0:1787830455.142055:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830455.142064:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830455.142110:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281013760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830455.142138:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830455.142141:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830455.142142:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830455.142145:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5004 00000100:00080000:3.0:1787830455.142147:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830455.142150:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830455.142159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830455.142167:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281013888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830455.142174:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830455.142178:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830455.142180:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830455.142185:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5004 00000100:00080000:3.0:1787830455.142189:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830455.142193:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830455.142199:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830455.142204:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281013760:192.168.203.124@tcp:250:kworker.0 00000800:00000100:0.0:1787830455.142281:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830455.142341:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830455/real 1787830455] req@ffff8ab3d0a41e00 x1874671281013760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830455.142358:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281013760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830455.142367:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830455.142371:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830455.142375:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281013760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830455.142384:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281013888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830455.142405:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830455.142418:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830455/real 1787830455] req@ffff8ab3d0a430c0 x1874671281013888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830455.142429:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281013888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830455.142436:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830455.142439:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830455.142442:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281013888:192.168.203.124@tcp:38:kworker.0 00010000:00010000:2.0:1787830455.142657:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830455.144548:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281014016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830455.144665:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281014016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830455.144742:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830455.144779:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830455.144785:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830455.144789:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830455.144795:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5004 00000100:00080000:3.0:1787830455.144802:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830455.144811:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830455/real 1787830455] req@ffff8ab3d0a421c0 x1874671281014016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830455.144812:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830455.144829:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830455.144831:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281014016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830455.144842:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830455.144844:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830455.144847:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281014016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830455.144854:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281014144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830455.144867:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830455.144871:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281014144:192.168.203.124@tcp:8:kworker.0 00000100:00080000:3.0:1787830455.144873:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830455.144877:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830455.144883:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5004 00000800:00000100:0.0:1787830455.145005:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830455.145050:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830455.145056:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830455.145062:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830455.145065:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830455/real 1787830455] req@ffff8ab3d0a430c0 x1874671281014144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830455.145075:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281014272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830455.145084:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281014144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830455.145103:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830455.145107:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830455.145111:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281014144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830455.145124:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281014272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830455.145145:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830455.145157:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830455/real 1787830455] req@ffff8ab3d0a41e00 x1874671281014272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830455.145168:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281014272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830455.145177:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830455.145180:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830455.145184:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281014272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830456.166361:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830457.190468:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830458.214281:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830459.238147:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830460.262043:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830460.262054:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830460.262059:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830460.262069:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5009 00000100:00080000:3.0:1787830460.262081:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830460.262089:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830460.262134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281014400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787830460.262158:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830460.262168:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830460.262173:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830460.262175:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830460.262179:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5009 00000100:00080000:3.0:1787830460.262183:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830460.262186:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830460.262195:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830460.262206:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281014528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830460.262215:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830460.262218:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830460.262220:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830460.262224:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5009 00000100:00080000:3.0:1787830460.262227:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830460.262230:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830460.262235:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830460.262244:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281014656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830460.262258:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830460.262261:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830460.262263:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830460.262266:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5009 00000100:00080000:3.0:1787830460.262269:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830460.262273:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830460.262277:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830460.262287:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281014784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830460.262295:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830460.262307:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830460.262310:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830460.262314:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5009 00000100:00080000:3.0:1787830460.262317:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830460.262321:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830460.262325:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830460.262335:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281014912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830460.262516:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281014400:192.168.203.124@tcp:250:kworker.0 00000800:00000100:0.0:1787830460.262574:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830460.262615:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830460/real 1787830460] req@ffff8ab3d0a40000 x1874671281014400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830460.262631:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281014400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830460.262641:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830460.262644:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830460.262648:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281014400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830460.262655:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281014528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830460.262676:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830460.262688:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830460/real 1787830460] req@ffff8ab3d0a43c00 x1874671281014528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830460.262699:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281014528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830460.262707:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830460.262710:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830460.262714:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281014528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830460.262719:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281014656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830460.262738:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830460.262749:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830460/real 1787830460] req@ffff8ab3d0a412c0 x1874671281014656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830460.262760:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281014656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830460.262778:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830460.262782:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830460.262785:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281014656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830460.262791:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281014784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830460.262810:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830460.262822:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830460/real 1787830460] req@ffff8ab3d0a43840 x1874671281014784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830460.262833:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281014784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830460.262843:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830460.262846:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830460.262849:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281014784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830460.262855:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281014912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830460.262872:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830460.262882:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830460/real 1787830460] req@ffff8ab3d0a43480 x1874671281014912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830460.262935:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281014912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830460.262944:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830460.262947:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830460.262950:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281014912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830461.286223:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830462.310495:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830463.334159:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830464.358138:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830465.381986:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830465.381996:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830465.382000:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830465.382012:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5015 00000100:00080000:3.0:1787830465.382021:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830465.382027:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:2.0:1787830465.382053:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830465.382062:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281015040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830465.382086:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830465.382090:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830465.382092:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830465.382096:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5015 00000100:00080000:3.0:1787830465.382099:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830465.382104:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830465.382113:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830465.382123:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281015168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830465.382127:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281015040:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787830465.382133:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830465.382136:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830465.382137:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830465.382141:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5015 00000100:00080000:3.0:1787830465.382144:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830465.382153:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830465.382158:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830465.382171:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281015296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830465.382180:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830465.382183:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830465.382185:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787830465.382192:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830465.382192:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5015 00000100:00080000:3.0:1787830465.382200:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830465.382205:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830465.382211:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830465.382222:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281015424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830465.382231:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830465.382234:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830465.382236:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830465.382239:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830465/real 1787830465] req@ffff8ab3d0a43480 x1874671281015040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830465.382243:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5015 00000100:00080000:3.0:1787830465.382250:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830465.382255:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830465.382260:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830465.382266:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281015040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830465.382271:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281015552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830465.382288:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830465.382294:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830465.382300:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281015040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830465.382322:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281015168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830465.382343:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830465.382355:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830465/real 1787830465] req@ffff8ab3d0a43840 x1874671281015168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830465.382366:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281015168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830465.382374:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830465.382377:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830465.382380:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281015168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830465.382386:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281015296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830465.382404:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830465.382414:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830465/real 1787830465] req@ffff8ab3d0a412c0 x1874671281015296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830465.382437:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281015296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830465.382452:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830465.382456:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830465.382459:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281015296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830465.382465:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281015424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830465.382484:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830465.382495:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830465/real 1787830465] req@ffff8ab3d0a43c00 x1874671281015424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830465.382505:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281015424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830465.382514:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830465.382518:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830465.382521:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281015424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830465.382526:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281015552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830465.382544:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830465.382555:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830465/real 1787830465] req@ffff8ab3d0a40000 x1874671281015552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830465.382565:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281015552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830465.382574:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830465.382577:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830465.382581:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281015552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830466.406100:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830467.430323:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830468.454275:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830469.478475:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830470.502094:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830470.502101:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830470.502105:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830470.502113:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5020 00000100:00080000:3.0:1787830470.502121:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830470.502127:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830470.502158:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281015680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830470.502193:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830470.502197:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830470.502199:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830470.502203:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5020 00000100:00080000:3.0:1787830470.502205:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830470.502208:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830470.502216:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830470.502224:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281015808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830470.502230:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830470.502232:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830470.502233:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830470.502236:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5020 00000100:00080000:3.0:1787830470.502237:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830470.502240:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830470.502242:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830470.502247:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281015936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830470.502252:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830470.502254:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830470.502255:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830470.502257:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5020 00000100:00080000:3.0:1787830470.502260:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830470.502262:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830470.502265:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830470.502271:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281016064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830470.502276:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830470.502278:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830470.502279:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830470.502281:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5020 00000100:00080000:3.0:1787830470.502283:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830470.502285:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830470.502287:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830470.502291:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281016192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830470.502379:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281015680:192.168.203.124@tcp:250:kworker.0 00000800:00000100:0.0:1787830470.502450:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830470.502496:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830470/real 1787830470] req@ffff8ab3d0a41e00 x1874671281015680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830470.502511:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281015680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830470.502520:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830470.502524:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830470.502527:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281015680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830470.502534:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281015808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830470.502556:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830470.502568:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830470/real 1787830470] req@ffff8ab3d0a430c0 x1874671281015808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830470.502582:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281015808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830470.502590:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830470.502593:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830470.502596:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281015808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830470.502602:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281015936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830470.502618:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830470.502625:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830470/real 1787830470] req@ffff8ab3d0a421c0 x1874671281015936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830470.502632:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281015936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830470.502639:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830470.502642:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830470.502645:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281015936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830470.502650:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281016064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830470.502666:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830470.502678:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830470/real 1787830470] req@ffff8ab3d0a40000 x1874671281016064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830470.502690:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281016064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830470.502701:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830470.502706:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830470.502709:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281016064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830470.502716:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281016192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830470.502734:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830470.502744:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830470/real 1787830470] req@ffff8ab3d0a43c00 x1874671281016192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830470.502762:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281016192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830470.502772:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830470.502775:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830470.502777:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281016192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830470.503165:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830471.526129:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830472.551322:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830473.574046:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830474.598202:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830475.622035:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830475.622046:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830475.622050:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830475.622061:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5025 00000100:00080000:0.0:1787830475.622072:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830475.622081:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830475.622116:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671281016320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830475.622141:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830475.622144:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830475.622146:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830475.622151:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5025 00000100:00080000:0.0:1787830475.622154:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830475.622158:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830475.622168:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830475.622181:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671281016448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830475.622197:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830475.622201:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830475.622202:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830475.622208:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5025 00000100:00080000:0.0:1787830475.622212:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830475.622216:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830475.622222:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830475.622235:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671281016576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830475.622246:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830475.622247:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281016320:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830475.622252:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830475.622255:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830475.622260:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5025 00000100:00080000:0.0:1787830475.622264:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830475.622270:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830475.622275:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830475.622303:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671281016704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830475.622314:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830475.622317:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830475.622319:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830475.622322:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5025 00010000:00010000:2.0:1787830475.622340:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830475.622389:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830475.622395:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:3.0:1787830475.622401:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:0.0:1787830475.622403:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830475.622418:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671281016832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830475.622453:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830475/real 1787830475] req@ffff8ab3f075f840 x1874671281016320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830475.622467:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671281016320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830475.622474:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830475.622477:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830475.622480:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281016320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830475.622489:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281016448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830475.622505:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830475.622516:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830475/real 1787830475] req@ffff8ab3f075ed00 x1874671281016448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830475.622527:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671281016448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830475.622534:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830475.622537:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830475.622540:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281016448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830475.622563:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281016576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830475.622582:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830475.622592:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830475/real 1787830475] req@ffff8ab3f075f480 x1874671281016576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830475.622604:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671281016576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830475.622611:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830475.622614:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830475.622616:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281016576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830475.622622:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281016704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830475.622640:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830475.622652:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830475/real 1787830475] req@ffff8ab3f075e580 x1874671281016704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830475.622662:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671281016704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830475.622669:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830475.622671:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830475.622673:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281016704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830475.622677:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281016832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830475.622690:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830475.622696:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830475/real 1787830475] req@ffff8ab3f075da40 x1874671281016832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830475.622703:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671281016832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830475.622708:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830475.622710:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830475.622711:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281016832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830476.646090:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830477.670179:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830478.694087:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830479.721113:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830480.742033:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830480.742043:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830480.742047:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830480.742059:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5030 00000100:00080000:2.0:1787830480.742070:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830480.742076:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830480.742112:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671281016960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830480.742190:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281016960:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787830480.742299:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830480.742305:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830480.742308:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830480.742313:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5030 00000100:00080000:2.0:1787830480.742330:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830480.742335:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830480.742345:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830480.742359:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671281017088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787830480.742404:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830480.742443:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281017088:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787830480.742520:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830480.742524:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830480.742526:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830480.742532:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5030 00000100:00080000:2.0:1787830480.742536:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830480.742541:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830480.742547:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830480.742560:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671281017216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830480.742576:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830480.742579:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830480.742580:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830480.742584:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5030 00000100:00080000:2.0:1787830480.742588:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830480.742591:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830480.742596:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830480.742605:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671281017344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830480.742625:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830480.742628:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830480.742630:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830480.742633:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5030 00000100:00080000:2.0:1787830480.742636:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830480.742639:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830480.742647:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830480.742657:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671281017472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830480.742850:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281017216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830480.742881:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281017344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830480.742930:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281017472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830481.766168:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830482.790237:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787830483.816464:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:3.0:1787830483.816527:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787830483.816543:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830483.816580:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830483.816592:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830483.816602:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830483.816610:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:0.0:1787830483.816721:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830480/real 1787830483] req@ffff8ab3f043a1c0 x1874671281017216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830483.816749:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671281017216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830483.816761:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830483.816766:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830483.816771:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281017216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830483.816779:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830480/real 1787830483] req@ffff8ab3f0439e00 x1874671281017344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830483.816791:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671281017344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830483.816801:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830483.816805:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830483.816808:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281017344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830483.816814:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830480/real 1787830483] req@ffff8ab3f043b840 x1874671281017472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830483.816827:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671281017472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830483.816837:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830483.816840:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830483.816843:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281017472:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830483.816850:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830480/real 1787830483] req@ffff8ab3f043a940 x1874671281017088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830483.816861:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671281017088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830483.816868:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830483.816871:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830483.816874:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281017088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830483.816881:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830480/real 1787830483] req@ffff8ab3f0439a40 x1874671281016960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830483.817012:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671281016960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830483.817019:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830483.817022:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830483.817025:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281016960:192.168.203.124@tcp:250:kworker.0 00010000:00010000:2.0:1787830483.818597:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830484.839157:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830485.862195:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830485.862205:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830485.862209:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830485.862219:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5035 00000100:00080000:0.0:1787830485.862229:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830485.862236:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830485.862273:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671281017600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830485.862361:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281017600:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830485.862420:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830485.862452:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830485.862456:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830485.862458:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830485.862462:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5035 00000100:00080000:0.0:1787830485.862466:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830485.862471:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830485.862480:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830485.862494:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671281017728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830485.864020:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830485/real 1787830485] req@ffff8ab3f075da40 x1874671281017600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830485.864044:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671281017600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830485.864054:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830485.864058:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830485.864062:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281017600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830485.864084:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281017728:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787830485.864575:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830485.864581:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830485.864584:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830485.864591:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5035 00000100:00080000:0.0:1787830485.864596:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830485.864603:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830485.864611:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830485.864633:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671281017856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830485.864645:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830485.864648:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830485.864650:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787830485.864685:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830485.864702:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830485/real 1787830485] req@ffff8ab3f075e580 x1874671281017728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830485.864715:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671281017728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830485.864724:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830485.864727:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830485.864730:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281017728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830485.864743:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281017856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830485.864761:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830485.864770:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830485/real 1787830485] req@ffff8ab3f075f480 x1874671281017856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830485.864782:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671281017856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830485.864789:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830485.864792:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830485.864795:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281017856:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787830485.866968:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5035 00000100:00080000:0.0:1787830485.866984:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830485.866996:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830485.867009:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830485.867041:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671281017984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830485.867071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830485.867076:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830485.867078:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830485.867083:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5035 00000100:00080000:0.0:1787830485.867087:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830485.867091:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830485.867096:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830485.867110:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671281018112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787830485.867126:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830485.867137:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281017984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830485.867192:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830485.867215:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830485/real 1787830485] req@ffff8ab3f075ed00 x1874671281017984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830485.867229:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671281017984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830485.867239:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830485.867242:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830485.867246:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281017984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830485.867251:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281018112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830485.867266:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830485.867274:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830485/real 1787830485] req@ffff8ab3f075f840 x1874671281018112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830485.867301:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671281018112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830485.867311:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830485.867314:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830485.867316:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281018112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830486.886316:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830487.912690:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830488.935053:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830489.960167:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830490.984385:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830490.986110:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830490.986121:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830490.986126:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830490.986138:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5040 00000100:00080000:2.0:1787830490.986151:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830490.986160:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830490.986208:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671281018240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830490.986372:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281018240:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787830490.986455:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830490.986460:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830490.986462:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830490.986466:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5040 00000100:00080000:2.0:1787830490.986471:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830490.986476:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830490.986488:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830490.986503:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671281018368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830490.986513:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830490.986517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830490.986518:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830490.986523:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5040 00000100:00080000:2.0:1787830490.986526:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830490.986530:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830490.986534:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830490.986544:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671281018496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830490.986553:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830490.986556:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830490.986558:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830490.986562:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5040 00000100:00080000:2.0:1787830490.986565:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830490.986570:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830490.986575:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830490.986585:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671281018624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830490.986593:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830490.986597:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830490.986599:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830490.986603:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5040 00000100:00080000:2.0:1787830490.986606:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830490.986610:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830490.986615:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830490.986624:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671281018752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830490.986741:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830490.986802:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830490/real 1787830490] req@ffff8ab3f043b480 x1874671281018240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830490.986823:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671281018240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830490.986834:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830490.986838:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830490.986842:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281018240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830490.986868:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281018368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830490.986927:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830490.986940:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830490/real 1787830490] req@ffff8ab3f0439a40 x1874671281018368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830490.986948:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671281018368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830490.986954:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830490.986956:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830490.986958:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281018368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830490.986962:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281018496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830490.986981:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830490.987466:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830490/real 1787830490] req@ffff8ab3f043a940 x1874671281018496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830490.987479:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671281018496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830490.987488:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830490.987492:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830490.987495:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281018496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830490.987501:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281018624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830490.987525:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830490.987535:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830490/real 1787830490] req@ffff8ab3f043b840 x1874671281018624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830490.987546:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671281018624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830490.987557:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830490.987560:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830490.987563:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281018624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830490.987569:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281018752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830490.987587:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830490.987597:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830490/real 1787830490] req@ffff8ab3f0439e00 x1874671281018752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830490.987608:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671281018752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830490.987618:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830490.987621:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830490.987624:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281018752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830492.006370:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830493.030655:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830494.054600:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830495.078266:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830496.102041:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830496.102049:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830496.102053:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830496.102063:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5045 00000100:00080000:0.0:1787830496.102073:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830496.102080:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830496.102127:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671281018880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830496.102155:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830496.102158:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830496.102161:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830496.102165:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5045 00000100:00080000:0.0:1787830496.102168:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830496.102173:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830496.102184:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00010000:00010000:2.0:1787830496.102189:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830496.102200:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671281019008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830496.102222:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830496.102227:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787830496.102227:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281018880:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830496.102231:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830496.102239:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5045 00000100:00080000:0.0:1787830496.102242:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830496.102246:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830496.102252:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830496.102266:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671281019136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830496.102276:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830496.102279:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830496.102281:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830496.102284:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5045 00000100:00080000:0.0:1787830496.102287:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787830496.102292:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830496.102295:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830496.102303:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830496.102315:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671281019264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830496.102324:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830496.102327:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830496.102329:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830496.102335:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5045 00000100:00080000:0.0:1787830496.102338:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830496.102340:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830496/real 1787830496] req@ffff8ab3f075f0c0 x1874671281018880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830496.102345:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830496.102355:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830496.102366:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671281018880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830496.102371:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671281019392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830496.102384:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830496.102391:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830496.102395:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281018880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830496.102419:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281019008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830496.102440:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830496.102453:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830496/real 1787830496] req@ffff8ab3f075cb40 x1874671281019008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830496.102465:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671281019008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830496.102474:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830496.102478:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830496.102481:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281019008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830496.102487:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281019136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830496.102505:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830496.102516:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830496/real 1787830496] req@ffff8ab3f075c000 x1874671281019136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830496.102527:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671281019136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830496.102536:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830496.102539:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830496.102542:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281019136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830496.102548:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281019264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830496.102566:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830496.102577:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830496/real 1787830496] req@ffff8ab3f075de00 x1874671281019264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830496.102587:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671281019264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830496.102596:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830496.102599:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830496.102602:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281019264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830496.102607:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281019392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830496.102624:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830496.102636:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830496/real 1787830496] req@ffff8ab3f075e940 x1874671281019392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830496.102647:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671281019392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830496.102656:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830496.102659:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830496.102661:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281019392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830497.126709:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830498.150177:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830499.174241:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830500.198699:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830501.222603:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830501.222712:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830501.222720:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830501.222724:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830501.222733:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5050 00000100:00080000:3.0:1787830501.222744:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830501.222751:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830501.222789:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281019520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830501.222826:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830501.222830:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830501.222832:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830501.222835:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5050 00000100:00080000:3.0:1787830501.222839:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830501.222842:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830501.222851:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830501.222863:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281019648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830501.222872:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830501.222875:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830501.222877:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830501.222881:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5050 00000100:00080000:3.0:1787830501.222915:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830501.223100:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281019520:192.168.203.124@tcp:250:kworker.0 00000800:00000100:0.0:1787830501.223159:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830501.223218:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830501/real 1787830501] req@ffff8ab3d0a412c0 x1874671281019520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830501.223234:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281019520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830501.223244:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830501.223248:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830501.223251:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281019520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830501.223258:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281019648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830501.223280:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830501.223305:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830501/real 1787830501] req@ffff8ab3d0a43840 x1874671281019648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830501.223317:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281019648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830501.223325:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830501.223329:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830501.223332:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281019648:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787830501.223421:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830501.223430:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830501.223445:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281019776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830501.223466:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830501.223470:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830501.223473:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830501.223477:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5050 00000100:00080000:3.0:1787830501.223481:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830501.223486:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830501.223491:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830501.223501:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281019904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830501.223510:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830501.223513:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830501.223515:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830501.223519:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5050 00000100:00080000:3.0:1787830501.223522:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830501.223526:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830501.223530:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830501.223538:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281020032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830501.223619:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281019776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830501.223644:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830501.223655:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830501/real 1787830501] req@ffff8ab3d0a43480 x1874671281019776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830501.223672:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281019776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830501.223680:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830501.223683:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830501.223686:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281019776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830501.223692:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281019904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830501.223707:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830501.223717:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830501/real 1787830501] req@ffff8ab3d0a43840 x1874671281019904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830501.223727:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281019904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830501.223736:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830501.223739:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830501.223742:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281019904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830501.223748:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281020032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830501.223763:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830501.223773:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830501/real 1787830501] req@ffff8ab3d0a412c0 x1874671281020032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830501.223783:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281020032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830501.223811:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830501.223814:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830501.223818:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281020032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830502.246953:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830503.271327:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830504.294185:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830505.318286:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830506.342099:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830506.342109:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830506.342114:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830506.342122:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5055 00000100:00080000:2.0:1787830506.342134:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830506.342155:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830506.342195:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671281020160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830506.342268:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281020160:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830506.342326:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830506.342357:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830506.342362:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830506.342364:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830506.342368:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5055 00000100:00080000:2.0:1787830506.342372:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830506.342377:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830506.342386:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830506.342401:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671281020288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787830506.345294:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830506.346327:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830506/real 1787830506] req@ffff8ab3f043a1c0 x1874671281020160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830506.346353:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671281020160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830506.346365:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830506.346370:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830506.346375:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281020160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830506.346406:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281020288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830506.346458:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830506.346474:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830506/real 1787830506] req@ffff8ab3f043bc00 x1874671281020288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830506.346487:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671281020288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830506.346495:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830506.346499:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830506.346502:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281020288:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787830506.346594:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830506.346600:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830506.346602:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830506.346608:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5055 00000100:00080000:2.0:1787830506.346613:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830506.346620:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830506.346630:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830506.346654:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671281020416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830506.346667:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830506.346671:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830506.346673:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830506.346677:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5055 00000100:00080000:2.0:1787830506.346680:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830506.346684:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830506.346688:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830506.346698:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671281020544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830506.346706:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830506.346710:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830506.346711:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830506.346715:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5055 00000100:00080000:2.0:1787830506.346718:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830506.346721:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830506.346726:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830506.346734:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671281020672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830506.346811:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281020416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830506.346834:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830506.346846:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830506/real 1787830506] req@ffff8ab3f04383c0 x1874671281020416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830506.346859:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671281020416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830506.346867:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830506.346870:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830506.346874:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281020416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830506.346879:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281020544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830506.346915:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830506.346927:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830506/real 1787830506] req@ffff8ab3f043ad00 x1874671281020544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830506.346938:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671281020544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830506.346947:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830506.346950:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830506.346953:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281020544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830506.346959:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281020672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830506.346978:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830506.346989:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830506/real 1787830506] req@ffff8ab3f04392c0 x1874671281020672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830506.346999:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671281020672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830506.347009:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830506.347012:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830506.347016:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281020672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830507.366114:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830508.390035:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830509.414175:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830510.438585:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830511.462794:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830511.463147:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830511.463153:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830511.463157:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830511.463167:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5061 00000100:00080000:2.0:1787830511.463177:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830511.463184:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830511.463213:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671281020800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830511.463247:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830511.463250:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830511.463252:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830511.463256:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5061 00000100:00080000:2.0:1787830511.463259:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830511.463263:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830511.463279:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830511.463290:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671281020928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830511.463300:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830511.463303:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830511.463305:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830511.463308:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5061 00000100:00080000:2.0:1787830511.463311:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830511.463315:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830511.463319:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830511.463328:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671281021056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830511.463337:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830511.463340:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830511.463342:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830511.463345:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5061 00000100:00080000:2.0:1787830511.463348:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830511.463352:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830511.463356:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830511.463364:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671281021184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830511.463373:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830511.463376:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830511.463377:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830511.463380:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5061 00000100:00080000:2.0:1787830511.463382:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830511.463386:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830511.463390:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830511.463397:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671281021312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830511.464414:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281020800:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830511.464572:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830511.464614:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830511/real 1787830511] req@ffff8ab3f0439680 x1874671281020800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830511.464638:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671281020800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830511.464649:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830511.464653:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830511.464657:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281020800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830511.464664:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281020928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830511.464687:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830511.464699:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830511/real 1787830511] req@ffff8ab3f043b0c0 x1874671281020928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830511.464711:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671281020928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830511.464720:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830511.464724:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830511.464727:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281020928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830511.464733:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281021056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830511.464750:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830511.464760:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830511/real 1787830511] req@ffff8ab3f043a580 x1874671281021056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830511.464772:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671281021056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830511.464787:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830511.464790:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830511.464793:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281021056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830511.464799:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281021184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830511.465932:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830511.465957:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830511/real 1787830511] req@ffff8ab3f0438f00 x1874671281021184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830511.465976:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671281021184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830511.465988:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830511.465993:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830511.465998:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281021184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830511.466007:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281021312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830511.466033:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830511.466047:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830511/real 1787830511] req@ffff8ab3f0438b40 x1874671281021312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830511.466059:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671281021312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830511.466068:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830511.466071:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830511.466075:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281021312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830512.486144:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830513.511003:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830514.537072:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830515.558125:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830516.582020:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830516.582031:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830516.582035:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830516.582045:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5066 00000100:00080000:2.0:1787830516.582055:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830516.582062:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830516.582107:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671281021440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830516.582133:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830516.582140:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830516.582142:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830516.582145:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5066 00000100:00080000:2.0:1787830516.582149:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830516.582153:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830516.582162:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830516.582175:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671281021568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830516.582184:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830516.582187:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830516.582188:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830516.582192:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5066 00000100:00080000:2.0:1787830516.582194:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830516.582199:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830516.582203:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830516.582212:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671281021696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830516.582220:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1787830516.582223:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830516.582233:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830516.582236:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830516.582241:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5066 00000100:00080000:2.0:1787830516.582244:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830516.582248:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830516.582253:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830516.582263:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671281021824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830516.582272:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830516.582276:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830516.582278:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830516.582282:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5066 00000100:00080000:2.0:1787830516.582286:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830516.582290:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830516.582295:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830516.582305:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671281021952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830516.582324:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281021440:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830516.582409:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830516.582467:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830516/real 1787830516] req@ffff8ab3f0438780 x1874671281021440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830516.582486:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671281021440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830516.582497:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830516.582502:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830516.582506:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281021440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830516.582513:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281021568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830516.582536:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830516.582548:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830516/real 1787830516] req@ffff8ab3f0438b40 x1874671281021568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830516.582559:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671281021568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830516.582566:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830516.582569:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830516.582572:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281021568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830516.582578:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281021696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830516.582597:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830516.582608:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830516/real 1787830516] req@ffff8ab3f0438f00 x1874671281021696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830516.582619:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671281021696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830516.582627:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830516.582631:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830516.582634:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281021696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830516.582640:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281021824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830516.582660:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830516.582671:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830516/real 1787830516] req@ffff8ab3f043a580 x1874671281021824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830516.582682:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671281021824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830516.582692:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830516.582696:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830516.582699:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281021824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830516.582733:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281021952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830516.582752:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830516.582763:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830516/real 1787830516] req@ffff8ab3f043b0c0 x1874671281021952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830516.582782:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671281021952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830516.582804:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830516.582808:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830516.582811:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281021952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830517.606176:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830518.630694:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830519.654560:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830520.678068:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830521.702076:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830521.702707:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830521.702714:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830521.702718:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830521.702727:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5071 00000100:00080000:1.0:1787830521.702737:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830521.702744:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830521.702782:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671281022080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830521.702809:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830521.702812:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830521.702814:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830521.702818:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5071 00000100:00080000:1.0:1787830521.702821:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830521.702825:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830521.702834:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830521.702845:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671281022208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830521.702860:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830521.702863:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830521.702865:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830521.702869:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5071 00000100:00080000:1.0:1787830521.703008:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830521.703013:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830521.703019:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830521.703044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671281022336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830521.703054:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830521.703058:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830521.703060:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830521.703064:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5071 00000100:00080000:1.0:1787830521.703067:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830521.703074:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281022080:192.168.203.124@tcp:250:kworker.0 00000100:00080000:1.0:1787830521.703075:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830521.703084:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830521.703098:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671281022464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830521.703109:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830521.703112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830521.703114:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830521.703118:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5071 00000100:00080000:1.0:1787830521.703132:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830521.703137:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:3.0:1787830521.703142:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:1.0:1787830521.703143:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830521.703158:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671281022592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830521.703192:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830521/real 1787830521] req@ffff8ab3d22fde00 x1874671281022080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830521.703208:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671281022080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830521.703216:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830521.703220:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830521.703223:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281022080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830521.703229:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281022208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830521.703250:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830521.703262:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830521/real 1787830521] req@ffff8ab3d22fe1c0 x1874671281022208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830521.703281:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671281022208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830521.703289:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830521.703291:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830521.703294:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281022208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830521.703300:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281022336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830521.703318:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830521.703328:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830521/real 1787830521] req@ffff8ab3d22fd2c0 x1874671281022336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830521.703339:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671281022336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830521.703347:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830521.703350:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830521.703353:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281022336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830521.703376:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281022464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830521.703395:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830521.703405:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830521/real 1787830521] req@ffff8ab3d22fc780 x1874671281022464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830521.703416:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671281022464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830521.703431:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830521.703434:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830521.703437:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281022464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830521.703442:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281022592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830521.703460:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830521.703471:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830521/real 1787830521] req@ffff8ab3d22fed00 x1874671281022592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830521.703481:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671281022592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830521.703491:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830521.703494:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830521.703497:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281022592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830522.726091:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830523.751366:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830524.774080:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830525.798220:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830526.822278:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830526.822289:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830526.822293:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830526.822304:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5076 00010000:00010000:1.0:1787830526.822312:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830526.822314:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830526.822321:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830526.822353:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671281022720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830526.822378:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830526.822381:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830526.822382:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830526.822385:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5076 00000100:00080000:0.0:1787830526.822387:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830526.822389:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830526.822397:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830526.822406:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671281022848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830526.822411:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830526.822413:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830526.822414:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830526.822417:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5076 00000100:00080000:0.0:1787830526.822418:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830526.822420:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830526.822423:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830526.822428:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671281022976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830526.822434:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830526.822435:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830526.822437:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830526.822439:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5076 00000100:00080000:0.0:1787830526.822440:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830526.822442:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830526.822445:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830526.822450:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671281023104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830526.822455:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830526.822457:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830526.822458:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830526.822460:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5076 00000100:00080000:0.0:1787830526.822462:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830526.822464:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830526.822466:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830526.822470:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671281023232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830526.822531:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281022720:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830526.822580:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830526.822614:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830526/real 1787830526] req@ffff8ab3f075cf00 x1874671281022720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830526.822625:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671281022720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830526.822632:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830526.822635:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830526.822638:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281022720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830526.822643:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281022848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830526.822654:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830526.822660:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830526/real 1787830526] req@ffff8ab3f075e940 x1874671281022848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830526.822667:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671281022848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830526.822671:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830526.822673:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830526.822675:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281022848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830526.822677:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281022976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830526.822686:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830526.822690:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830526/real 1787830526] req@ffff8ab3f075de00 x1874671281022976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830526.822697:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671281022976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830526.822701:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830526.822703:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830526.822704:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281022976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830526.822707:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281023104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830526.822728:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830526.822738:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830526/real 1787830526] req@ffff8ab3f075c000 x1874671281023104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830526.822748:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671281023104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830526.822757:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830526.822760:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830526.822763:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281023104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830526.822768:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281023232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830526.822784:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830526.822792:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830526/real 1787830526] req@ffff8ab3f075cb40 x1874671281023232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830526.822799:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671281023232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830526.822804:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830526.822806:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830526.822808:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281023232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830527.846227:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830528.870032:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830529.894523:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830530.918470:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830531.942029:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830531.942039:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830531.942043:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830531.942054:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5081 00000100:00080000:0.0:1787830531.942065:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830531.942073:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830531.942113:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671281023360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830531.942147:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830531.942151:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830531.942153:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830531.942156:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5081 00000100:00080000:0.0:1787830531.942160:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830531.942165:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830531.942175:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830531.942187:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671281023488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830531.942196:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830531.942199:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830531.942201:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830531.942204:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5081 00000100:00080000:0.0:1787830531.942207:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830531.942212:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830531.942216:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830531.942226:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671281023616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830531.942234:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830531.942237:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830531.942239:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830531.942243:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5081 00000100:00080000:0.0:1787830531.942246:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830531.942250:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830531.942255:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830531.942265:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671281023744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830531.942273:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830531.942276:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830531.942278:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830531.942281:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5081 00000100:00080000:0.0:1787830531.942284:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830531.942287:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830531.942291:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830531.942299:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671281023872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787830531.942556:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830531.942838:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281023360:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830531.944026:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830531.944076:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830531/real 1787830531] req@ffff8ab3f075f0c0 x1874671281023360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830531.944094:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671281023360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830531.944107:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830531.944111:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830531.944115:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281023360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830531.944125:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281023488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830531.944154:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830531.944178:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830531/real 1787830531] req@ffff8ab3f075f840 x1874671281023488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830531.944191:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671281023488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830531.944200:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830531.944204:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830531.944207:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281023488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830531.944215:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281023616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830531.944244:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830531.944255:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830531/real 1787830531] req@ffff8ab3f075ed00 x1874671281023616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830531.944266:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671281023616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830531.944274:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830531.944278:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830531.944281:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281023616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830531.944287:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281023744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830531.944304:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830531.944313:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830531/real 1787830531] req@ffff8ab3f075f480 x1874671281023744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830531.944323:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671281023744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830531.944332:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830531.944335:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830531.944338:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281023744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830531.944343:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281023872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830531.944359:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830531.944368:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830531/real 1787830531] req@ffff8ab3f075e580 x1874671281023872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830531.944378:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671281023872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830531.944387:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830531.944390:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830531.944393:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281023872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830532.966041:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830533.990874:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830535.014073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830536.038371:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830537.062055:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830537.062215:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830537.062221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830537.062226:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830537.062237:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5086 00000100:00080000:3.0:1787830537.062249:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830537.062256:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830537.062304:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281024000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830537.062373:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281024000:192.168.203.124@tcp:250:kworker.0 00000800:00000100:0.0:1787830537.062459:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830537.062493:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830537.062497:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830537.062500:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830537.062504:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5086 00000100:00080000:3.0:1787830537.062508:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830537.062512:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830537.062524:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830537.062539:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281024128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830537.062624:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830537/real 1787830537] req@ffff8ab3d0a43c00 x1874671281024000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830537.062645:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281024000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830537.062667:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830537.062672:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830537.062676:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281024000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830537.062701:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281024128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830537.062726:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830537.062740:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830537/real 1787830537] req@ffff8ab3d0a40000 x1874671281024128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830537.062752:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281024128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830537.062766:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830537.062770:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830537.062773:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281024128:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787830537.063026:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830537.063031:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830537.063033:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830537.063038:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5086 00000100:00080000:3.0:1787830537.063042:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830537.063046:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830537.063052:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830537.063065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281024256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830537.063094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830537.063097:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830537.063099:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830537.063103:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5086 00000100:00080000:3.0:1787830537.063107:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830537.063111:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830537.063116:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830537.063128:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281024384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830537.063137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830537.063140:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830537.063142:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830537.063146:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5086 00000100:00080000:3.0:1787830537.063149:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830537.063153:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830537.063157:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830537.063167:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281024512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830537.063221:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281024256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830537.063250:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830537.063264:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830537/real 1787830537] req@ffff8ab3d0a421c0 x1874671281024256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830537.063276:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281024256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830537.063284:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830537.063288:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830537.063290:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281024256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830537.063296:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281024384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830537.063316:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830537.063328:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830537/real 1787830537] req@ffff8ab3d0a430c0 x1874671281024384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830537.063339:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281024384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830537.063349:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830537.063352:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830537.063356:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281024384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830537.063361:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281024512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830537.063380:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830537.063392:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830537/real 1787830537] req@ffff8ab3d0a41e00 x1874671281024512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830537.063403:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281024512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830537.063412:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830537.063415:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830537.063418:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281024512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830538.086323:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830539.110154:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830540.134276:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830541.158469:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830542.182054:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830542.182064:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830542.182068:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830542.182077:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5091 00000100:00080000:2.0:1787830542.182087:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830542.182094:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830542.182129:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671281024640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830542.182153:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830542.182156:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830542.182158:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830542.182376:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281024640:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830542.182440:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830542.182482:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830542/real 1787830542] req@ffff8ab3f0439680 x1874671281024640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830542.182498:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671281024640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830542.182517:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830542.182521:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830542.182525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281024640:192.168.203.124@tcp:250:kworker.0 00010000:00010000:0.0:1787830542.182648:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830542.182856:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5091 00000100:00080000:2.0:1787830542.182861:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830542.182866:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830542.183112:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830542.183131:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671281024768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830542.183151:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830542.183154:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830542.183157:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830542.183161:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5091 00000100:00080000:2.0:1787830542.183329:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830542.183336:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830542.183342:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830542.183362:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671281024896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830542.183371:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830542.183374:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830542.183376:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830542.183380:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5091 00000100:00080000:2.0:1787830542.183384:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830542.183388:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830542.183393:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830542.183402:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671281025024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830542.183410:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830542.183413:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830542.183415:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830542.183429:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5091 00000100:00080000:2.0:1787830542.183432:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830542.183436:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830542.183440:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830542.183449:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671281025152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830542.183519:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281024768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830542.183549:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830542.183564:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830542/real 1787830542] req@ffff8ab3f04392c0 x1874671281024768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830542.183577:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671281024768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830542.183595:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830542.183599:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830542.183602:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281024768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830542.183608:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281024896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830542.183625:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830542.183865:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830542/real 1787830542] req@ffff8ab3f043ad00 x1874671281024896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830542.183995:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671281024896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830542.184003:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830542.184006:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830542.184010:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281024896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830542.184016:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281025024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830542.184038:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830542.184049:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830542/real 1787830542] req@ffff8ab3f04383c0 x1874671281025024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830542.184059:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671281025024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830542.184067:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830542.184069:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830542.184072:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281025024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830542.184077:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281025152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830542.184095:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830542.184105:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830542/real 1787830542] req@ffff8ab3f043bc00 x1874671281025152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830542.184115:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671281025152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830542.184124:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830542.184127:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830542.184130:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281025152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830543.206629:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830544.230699:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830545.255052:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830546.278719:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830547.302026:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830547.302037:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830547.302041:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830547.302052:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5096 00000100:00080000:2.0:1787830547.302062:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830547.302068:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830547.302107:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671281025280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830547.302136:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830547.302140:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830547.302142:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830547.302146:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5096 00000100:00080000:2.0:1787830547.302149:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830547.302153:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830547.302163:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830547.302175:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671281025408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830547.302184:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830547.302187:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830547.302189:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830547.302192:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5096 00000100:00080000:2.0:1787830547.302195:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830547.302199:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830547.302203:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830547.302213:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671281025536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830547.302222:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830547.302225:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830547.302226:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830547.302230:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5096 00000100:00080000:2.0:1787830547.302233:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830547.302237:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830547.302241:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830547.302252:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671281025664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830547.302261:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830547.302264:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830547.302266:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830547.302269:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5096 00000100:00080000:2.0:1787830547.302272:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830547.302275:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830547.302280:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830547.302289:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671281025792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830547.302519:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281025280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830547.302619:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281025408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830547.302646:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281025536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830547.302670:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281025664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830547.302769:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281025792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830547.304103:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830548.326177:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830549.350162:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830550.374291:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787830550.374356:0:10942:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787830550.374394:0:10942:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787830550.374410:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787830550.374447:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787830550.374457:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787830550.374464:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787830550.374472:0:10942:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:3.0:1787830550.374543:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830547/real 1787830550] req@ffff8ab3f043a1c0 x1874671281025280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830550.374566:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671281025280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830550.374577:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830550.374582:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830550.374586:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281025280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830550.374593:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830547/real 1787830550] req@ffff8ab3f0439e00 x1874671281025408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830550.374604:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671281025408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830550.374611:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830550.374615:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830550.374618:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281025408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830550.374624:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830547/real 1787830550] req@ffff8ab3f043b840 x1874671281025536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830550.374635:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671281025536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830550.374648:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830550.374651:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830550.374655:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281025536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830550.374661:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830547/real 1787830550] req@ffff8ab3f043a940 x1874671281025664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830550.374671:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671281025664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830550.374680:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830550.374683:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830550.374686:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281025664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830550.374692:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830547/real 1787830550] req@ffff8ab3f0439a40 x1874671281025792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830550.374702:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671281025792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830550.374710:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830550.374714:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830550.374717:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281025792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830551.398324:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830552.424791:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830552.428484:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830552.428497:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830552.428501:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830552.428511:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5102 00000100:00080000:3.0:1787830552.428523:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830552.428529:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830552.428568:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281025920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830552.428607:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830552.428611:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830552.428613:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830552.428617:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5102 00000100:00080000:3.0:1787830552.428620:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830552.428625:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830552.428634:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830552.428646:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281026048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830552.428655:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830552.428658:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830552.428660:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830552.428665:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5102 00000100:00080000:3.0:1787830552.428668:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830552.428672:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830552.428677:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830552.428687:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281026176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830552.428695:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830552.428699:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830552.428700:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830552.428704:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5102 00000100:00080000:3.0:1787830552.428707:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830552.428711:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830552.428715:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830552.428725:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281026304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830552.428734:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830552.428736:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830552.428738:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830552.428743:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5102 00000100:00080000:3.0:1787830552.428745:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830552.428749:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830552.428752:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830552.428760:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281026432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830552.428909:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281025920:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830552.428970:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830552.429063:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830552/real 1787830552] req@ffff8ab3d0a41e00 x1874671281025920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830552.429080:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281025920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830552.429088:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830552.429354:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830552.429358:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281025920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830552.429367:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281026048:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830552.429392:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830552.429406:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830552/real 1787830552] req@ffff8ab3d0a430c0 x1874671281026048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830552.429419:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281026048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830552.429426:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830552.429429:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830552.429432:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281026048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830552.429438:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281026176:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830552.429456:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830552.429466:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830552/real 1787830552] req@ffff8ab3d0a421c0 x1874671281026176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830552.429476:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281026176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830552.429489:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830552.429492:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830552.429495:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281026176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830552.429501:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281026304:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830552.429520:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830552.429530:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830552/real 1787830552] req@ffff8ab3d0a40000 x1874671281026304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830552.429540:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281026304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830552.429549:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830552.429552:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830552.429555:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281026304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830552.429561:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281026432:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830552.429578:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830552.429589:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830552/real 1787830552] req@ffff8ab3d0a43c00 x1874671281026432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830552.429600:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281026432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830552.429610:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830552.429612:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830552.429616:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281026432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830553.447120:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830554.470199:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830555.494173:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830556.518235:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830557.542438:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830557.546883:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830557.546953:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830557.546958:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830557.546967:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5107 00000100:00080000:0.0:1787830557.546977:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830557.546984:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830557.547020:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671281026560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830557.547056:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830557.547060:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830557.547061:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830557.547065:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5107 00000100:00080000:0.0:1787830557.547068:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830557.547073:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830557.547082:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830557.547097:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281026560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830557.547100:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671281026688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830557.547112:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830557.547115:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830557.547117:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830557.547121:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5107 00000100:00080000:0.0:1787830557.547124:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830557.547128:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830557.547133:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830557.547143:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671281026816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830557.547152:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830557.547155:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830557.547157:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830557.547160:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5107 00000100:00080000:0.0:1787830557.547163:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830557.547167:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830557.547171:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830557.547180:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671281026944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830557.547189:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830557.547192:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830557.547194:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830557.547197:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5107 00000100:00080000:0.0:1787830557.547200:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830557.547203:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830557.547207:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830557.547216:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671281027072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830557.547400:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830557.547460:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830557/real 1787830557] req@ffff8ab3f075da40 x1874671281026560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830557.547576:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671281026560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830557.547588:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830557.547591:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830557.547595:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281026560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830557.547615:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281026688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830557.547636:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830557.547648:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830557/real 1787830557] req@ffff8ab3f075e580 x1874671281026688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830557.547659:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671281026688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830557.547667:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830557.547670:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830557.547673:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281026688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830557.547678:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281026816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830557.547696:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830557.547707:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830557/real 1787830557] req@ffff8ab3f075f480 x1874671281026816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830557.547718:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671281026816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830557.547725:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830557.547727:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830557.547730:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281026816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830557.547736:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281026944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830557.547753:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830557.548131:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830557/real 1787830557] req@ffff8ab3f075ed00 x1874671281026944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830557.548143:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671281026944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830557.548287:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830557.548292:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830557.548296:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281026944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830557.548302:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281027072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830557.548323:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830557.548333:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830557/real 1787830557] req@ffff8ab3f075f840 x1874671281027072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830557.548343:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671281027072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830557.548353:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830557.548355:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830557.548358:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281027072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830558.567339:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830559.592050:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830560.614239:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830561.638193:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830562.662037:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830562.662050:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830562.662056:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830562.662067:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5112 00000100:00080000:0.0:1787830562.662079:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830562.662089:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830562.662138:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671281027200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830562.662177:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830562.662181:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830562.662183:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830562.662186:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5112 00000100:00080000:0.0:1787830562.662190:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830562.662194:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830562.662205:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830562.662218:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671281027328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830562.662227:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830562.662230:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830562.662232:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830562.662235:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5112 00000100:00080000:0.0:1787830562.662238:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830562.662242:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830562.662246:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830562.662256:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671281027456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830562.662264:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830562.662268:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830562.662270:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830562.662276:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5112 00000100:00080000:0.0:1787830562.662279:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830562.662284:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830562.662288:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830562.662298:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671281027584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830562.662306:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830562.662309:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830562.662311:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830562.662315:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5112 00000100:00080000:0.0:1787830562.662318:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830562.662322:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830562.662326:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830562.662335:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671281027712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787830562.662425:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830562.662488:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281027200:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830562.662560:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830562.662603:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830562/real 1787830562] req@ffff8ab3f075f0c0 x1874671281027200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830562.662619:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671281027200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830562.662629:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830562.662633:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830562.662636:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281027200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830562.662643:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281027328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830562.662662:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830562.662674:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830562/real 1787830562] req@ffff8ab3f075cb40 x1874671281027328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830562.662684:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671281027328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830562.662691:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830562.662694:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830562.662696:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281027328:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830562.662702:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281027456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830562.662720:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830562.662730:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830562/real 1787830562] req@ffff8ab3f075c000 x1874671281027456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830562.662739:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671281027456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830562.662764:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830562.662767:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830562.662769:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281027456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830562.662779:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281027584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830562.662798:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830562.662808:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830562/real 1787830562] req@ffff8ab3f075de00 x1874671281027584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830562.662819:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671281027584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830562.662828:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830562.662830:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830562.662833:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281027584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830562.662838:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281027712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830562.662857:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830562.662868:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830562/real 1787830562] req@ffff8ab3f075e940 x1874671281027712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830562.662878:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671281027712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830562.662916:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830562.662920:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830562.662923:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281027712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830563.686205:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830564.710131:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830565.734142:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830566.758139:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830567.782048:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830567.782112:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830567.782118:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830567.782122:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830567.782132:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5117 00000100:00080000:2.0:1787830567.782142:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830567.782149:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830567.782187:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671281027840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830567.782218:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830567.782221:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830567.782223:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830567.782227:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5117 00000100:00080000:2.0:1787830567.782231:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830567.782234:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830567.782244:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830567.782258:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671281027968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830567.782266:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830567.782269:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830567.782270:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830567.782274:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5117 00000100:00080000:2.0:1787830567.782277:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830567.782281:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830567.782286:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830567.782296:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671281028096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830567.782304:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830567.782307:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830567.782308:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830567.782312:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5117 00000100:00080000:2.0:1787830567.782315:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830567.782319:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830567.782324:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830567.782333:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671281028224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830567.782341:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830567.782345:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830567.782346:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830567.782350:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5117 00000100:00080000:2.0:1787830567.782353:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830567.782357:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830567.782361:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830567.782370:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671281028352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830567.782443:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281027840:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830567.782498:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830567.782543:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830567/real 1787830567] req@ffff8ab3f043b480 x1874671281027840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830567.782559:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671281027840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830567.782569:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830567.782573:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830567.782577:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281027840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830567.782584:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281027968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830567.782604:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830567.782616:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830567/real 1787830567] req@ffff8ab3f0439a40 x1874671281027968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830567.782625:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671281027968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830567.782637:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830567.782640:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830567.782643:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281027968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830567.782648:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281028096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830567.782677:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830567.782687:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830567/real 1787830567] req@ffff8ab3f043a940 x1874671281028096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830567.782697:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671281028096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830567.782704:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830567.782707:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830567.782721:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281028096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830567.782727:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281028224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830567.782743:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830567.782753:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830567/real 1787830567] req@ffff8ab3f043b840 x1874671281028224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830567.782762:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671281028224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830567.782770:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830567.782773:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830567.782776:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281028224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830567.782781:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281028352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830567.782795:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830567.782804:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830567/real 1787830567] req@ffff8ab3f0439e00 x1874671281028352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830567.782813:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671281028352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830567.782822:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830567.782825:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830567.782828:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281028352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830568.806174:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830569.830186:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830570.854268:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830571.878759:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830572.902130:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830572.902143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830572.902150:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830572.902162:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5122 00000100:00080000:3.0:1787830572.902175:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830572.902185:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830572.902239:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281028480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830572.902269:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830572.902274:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830572.902278:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830572.902284:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5122 00000100:00080000:3.0:1787830572.902289:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830572.902294:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787830572.902298:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:3.0:1787830572.902308:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830572.902310:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281028480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830572.902331:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281028608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830572.902345:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830572.902348:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830572.902350:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830572.902353:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5122 00000100:00080000:3.0:1787830572.902370:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830572.902374:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787830572.902381:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787830572.902385:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830572.902407:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281028736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830572.902419:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830572.902423:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830572/real 1787830572] req@ffff8ab3d0a412c0 x1874671281028480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830572.902425:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830572.902428:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830572.902445:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5122 00000100:00080000:3.0:1787830572.902458:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830572.902467:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830572.902468:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281028480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787830572.902483:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00080000:1.0:1787830572.902487:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830572.902491:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830572.902496:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281028480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830572.902500:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281028864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830572.902513:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830572.902515:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830572.902517:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830572.902521:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5122 00000100:00080000:3.0:1787830572.902524:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830572.902526:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830572.902529:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830572.902535:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281028608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830572.902538:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281028992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830572.902584:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830572.902600:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830572/real 1787830572] req@ffff8ab3d0a43840 x1874671281028608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830572.902612:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281028608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830572.902621:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830572.902625:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830572.902628:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281028608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830572.902635:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281028736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830572.902653:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830572.902665:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830572/real 1787830572] req@ffff8ab3d0a43480 x1874671281028736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830572.902689:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281028736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830572.902706:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830572.902710:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830572.902713:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281028736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830572.902720:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281028864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830572.902739:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830572.902750:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830572/real 1787830572] req@ffff8ab3d0a43c00 x1874671281028864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830572.902837:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281028864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830572.902846:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830572.902848:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830572.902851:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281028864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830572.902864:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281028992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830572.902880:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830572.902907:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830572/real 1787830572] req@ffff8ab3d0a40000 x1874671281028992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830572.902914:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281028992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830572.902920:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830572.902922:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830572.902924:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281028992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830573.926139:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830574.950107:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830575.974324:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830576.998202:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830578.022002:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830578.022014:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830578.022019:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830578.022030:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5127 00000100:00080000:2.0:1787830578.022041:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830578.022049:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830578.022083:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671281029120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830578.022121:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830578.022125:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830578.022127:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830578.022131:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5127 00000100:00080000:2.0:1787830578.022135:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830578.022139:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830578.022147:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830578.022158:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671281029248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830578.022167:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830578.022170:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830578.022172:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830578.022175:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5127 00000100:00080000:2.0:1787830578.022178:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830578.022183:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830578.022187:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00010000:00010000:0.0:1787830578.022189:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830578.022203:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671281029376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830578.022212:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830578.022215:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830578.022217:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830578.022220:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5127 00000100:00080000:2.0:1787830578.022223:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830578.022227:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830578.022230:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830578.022238:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671281029504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830578.022246:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830578.022249:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830578.022251:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830578.022254:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5127 00000100:00080000:2.0:1787830578.022257:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830578.022261:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830578.022265:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830578.022274:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671281029632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830578.022315:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281029120:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830578.022371:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830578.022412:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830578/real 1787830578] req@ffff8ab3f043a1c0 x1874671281029120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830578.022428:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671281029120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830578.022438:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830578.022442:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830578.022445:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281029120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830578.022453:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281029248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830578.022487:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830578.022499:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830578/real 1787830578] req@ffff8ab3f043bc00 x1874671281029248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830578.022510:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671281029248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830578.022518:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830578.022521:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830578.022524:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281029248:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830578.022530:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281029376:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830578.022548:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830578.022558:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830578/real 1787830578] req@ffff8ab3f04383c0 x1874671281029376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830578.022568:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671281029376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830578.022575:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830578.022578:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830578.022581:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281029376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830578.022586:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281029504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830578.022604:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830578.022615:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830578/real 1787830578] req@ffff8ab3f043ad00 x1874671281029504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830578.022625:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671281029504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830578.022633:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830578.022636:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830578.022650:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281029504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830578.022656:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281029632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830578.022674:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830578.022685:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830578/real 1787830578] req@ffff8ab3f04392c0 x1874671281029632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830578.022695:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671281029632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830578.022704:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830578.022706:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830578.022710:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281029632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830579.046314:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830580.070685:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830581.094295:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830582.118231:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830583.142018:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830583.142027:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830583.142030:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830583.142040:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5132 00000100:00080000:2.0:1787830583.142050:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830583.142057:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830583.142091:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671281029760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830583.142115:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830583.142119:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830583.142121:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830583.142124:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5132 00000100:00080000:2.0:1787830583.142128:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830583.142132:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:0.0:1787830583.142139:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:2.0:1787830583.142141:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830583.142155:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671281029888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830583.142163:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830583.142166:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830583.142168:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830583.142172:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5132 00000100:00080000:2.0:1787830583.142175:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830583.142178:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830583.142183:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830583.142191:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671281030016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830583.142200:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830583.142203:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830583.142205:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830583.142208:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5132 00000100:00080000:2.0:1787830583.142210:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830583.142214:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830583.142219:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830583.142227:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671281030144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830583.142235:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830583.142238:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830583.142239:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830583.142243:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5132 00000100:00080000:2.0:1787830583.142245:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830583.142249:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830583.142253:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830583.142261:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671281030272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830583.142981:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281029760:192.168.203.124@tcp:250:kworker.0 00000800:00000100:3.0:1787830583.143036:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830583.143080:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830583/real 1787830583] req@ffff8ab3f0439680 x1874671281029760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830583.143092:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671281029760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830583.143100:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830583.143102:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830583.143104:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281029760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830583.143109:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281029888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830583.143122:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830583.143129:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830583/real 1787830583] req@ffff8ab3f043b0c0 x1874671281029888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830583.143136:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671281029888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830583.143141:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830583.143142:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830583.143144:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281029888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830583.143147:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281030016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830583.143156:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830583.143160:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830583/real 1787830583] req@ffff8ab3f043a580 x1874671281030016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830583.143166:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671281030016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830583.143171:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830583.143172:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830583.143175:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281030016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830583.143178:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281030144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830583.143186:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830583.143191:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830583/real 1787830583] req@ffff8ab3f0438f00 x1874671281030144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830583.143196:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671281030144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830583.143202:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830583.143203:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830583.143205:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281030144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830583.143208:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281030272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830583.143215:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830583.143220:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830583/real 1787830583] req@ffff8ab3f0438b40 x1874671281030272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830583.143227:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671281030272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830583.143231:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830583.143233:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830583.143235:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281030272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830584.167203:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830585.190466:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830586.214206:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830587.238076:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830588.262490:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830588.262500:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830588.262504:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830588.262514:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5137 00000100:00080000:2.0:1787830588.262524:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830588.262530:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830588.262576:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671281030400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830588.262604:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830588.262608:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830588.262610:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830588.262614:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5137 00000100:00080000:2.0:1787830588.262617:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830588.262621:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830588.262637:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830588.262649:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671281030528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830588.262657:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830588.262659:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830588.262660:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830588.262663:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5137 00000100:00080000:2.0:1787830588.262666:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830588.262670:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830588.262675:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830588.262685:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671281030656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830588.262695:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281030400:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787830588.262698:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830588.262714:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830588.262716:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830588.262721:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5137 00000100:00080000:2.0:1787830588.262725:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830588.262730:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830588.262735:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830588.262745:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671281030784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830588.262752:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830588.262754:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830588.262755:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787830588.262760:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830588.262761:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5137 00000100:00080000:2.0:1787830588.262766:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830588.262770:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830588.262774:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830588.262781:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671281030912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830588.262821:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830588/real 1787830588] req@ffff8ab3f0438780 x1874671281030400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830588.262841:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671281030400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830588.262852:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830588.262856:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830588.262860:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281030400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830588.262868:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281030528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830588.262909:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830588.262923:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830588/real 1787830588] req@ffff8ab3f0438b40 x1874671281030528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830588.262935:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671281030528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830588.262944:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830588.262948:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830588.262953:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281030528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830588.262977:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281030656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:3.0:1787830588.263000:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830588.263012:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830588/real 1787830588] req@ffff8ab3f0438f00 x1874671281030656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830588.263024:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671281030656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830588.263040:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830588.263044:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830588.263048:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281030656:192.168.203.124@tcp:38:kworker.0 00010000:00010000:0.0:1787830588.263055:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830588.263056:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281030784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830588.263081:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830588.263093:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830588/real 1787830588] req@ffff8ab3f043a580 x1874671281030784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830588.263104:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671281030784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830588.263113:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830588.263117:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830588.263119:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281030784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:3.0:1787830588.263125:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281030912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:3.0:1787830588.263243:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830588.263255:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830588/real 1787830588] req@ffff8ab3f043b0c0 x1874671281030912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830588.263266:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671281030912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830588.263276:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830588.263279:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:3.0:1787830588.263283:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281030912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830589.286620:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830590.311254:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830591.334747:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830592.360322:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830593.382093:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830593.382103:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830593.382107:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830593.382117:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5143 00000100:00080000:3.0:1787830593.382127:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830593.382135:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830593.382173:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281031040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830593.382198:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830593.382202:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830593.382204:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830593.382207:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5143 00000100:00080000:3.0:1787830593.382211:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830593.382215:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830593.382224:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830593.382236:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281031168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830593.382245:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830593.382248:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830593.382250:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830593.382255:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5143 00000100:00080000:3.0:1787830593.382260:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830593.382264:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830593.382269:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830593.382282:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281031040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830593.382282:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281031296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830593.382297:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830593.382300:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830593.382302:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830593.382306:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5143 00000100:00080000:3.0:1787830593.382309:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830593.382314:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830593.382320:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830593.382333:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281031424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830593.382336:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830593.382347:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830593.382351:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830593.382353:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830593.382358:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5143 00000100:00080000:3.0:1787830593.382362:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830593.382367:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830593.382372:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830593.382383:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281031552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830593.382391:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830593/real 1787830593] req@ffff8ab3d0a421c0 x1874671281031040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830593.382426:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281031040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830593.382436:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830593.382439:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830593.382442:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281031040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830593.382448:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281031168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830593.382464:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830593.382473:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830593/real 1787830593] req@ffff8ab3d0a430c0 x1874671281031168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830593.382481:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281031168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830593.382485:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830593.382487:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830593.382489:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281031168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830593.382505:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281031296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830593.382515:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830593.382523:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830593/real 1787830593] req@ffff8ab3d0a41e00 x1874671281031296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830593.382544:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281031296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830593.382552:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830593.382555:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830593.382558:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281031296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830593.382564:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281031424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830593.382582:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830593.382591:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830593/real 1787830593] req@ffff8ab3d0a40000 x1874671281031424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830593.382601:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281031424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830593.382610:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830593.382613:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830593.382616:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281031424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830593.382622:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281031552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830593.382641:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830593.382663:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830593/real 1787830593] req@ffff8ab3d0a43c00 x1874671281031552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830593.382677:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281031552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830593.382686:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830593.382689:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830593.382692:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281031552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830593.383597:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830594.406168:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830595.430124:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830596.454252:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830597.478569:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830598.502045:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830598.502055:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830598.502059:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830598.502069:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5148 00000100:00080000:3.0:1787830598.502080:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830598.502087:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830598.502123:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281031680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830598.502163:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830598.502167:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830598.502169:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830598.502173:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5148 00000100:00080000:3.0:1787830598.502177:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830598.502182:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830598.502192:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830598.502205:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281031808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830598.502215:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281031680:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787830598.502219:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830598.502224:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830598.502226:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830598.502230:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5148 00000100:00080000:3.0:1787830598.502235:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830598.502239:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830598.502243:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830598.502254:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281031936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830598.502263:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830598.502267:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830598.502269:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830598.502272:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5148 00000800:00000100:0.0:1787830598.502302:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00010000:00010000:2.0:1787830598.502365:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830598.505664:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830598.505683:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830598.505696:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830598.505741:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281032064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830598.505758:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830598.505761:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830598.505764:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830598.505769:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5148 00000100:00080000:3.0:1787830598.505774:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830598.505778:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830598.505782:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830598.505795:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281032192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830598.506587:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830598/real 1787830598] req@ffff8ab3d0a43480 x1874671281031680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830598.506607:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281031680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830598.506617:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830598.506621:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830598.506675:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281031680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830598.506706:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281031808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830598.506759:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830598.506774:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830598/real 1787830598] req@ffff8ab3d0a43840 x1874671281031808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830598.506789:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281031808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830598.506796:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830598.506799:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830598.506802:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281031808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830598.506808:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281031936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830598.506826:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830598.506835:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830598/real 1787830598] req@ffff8ab3d0a412c0 x1874671281031936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830598.506845:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281031936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830598.506852:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830598.506854:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830598.506857:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281031936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830598.506862:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281032064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830598.506879:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830598.506905:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830598/real 1787830598] req@ffff8ab3d0a43c00 x1874671281032064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830598.506915:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281032064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830598.506930:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830598.506933:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830598.506936:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281032064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830598.506941:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281032192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830598.506957:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830598.506975:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830598/real 1787830598] req@ffff8ab3d0a40000 x1874671281032192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830598.506987:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281032192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830598.506995:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830598.506998:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830598.507001:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281032192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830599.526953:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830600.550065:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830601.574276:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830602.599579:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830603.622008:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830603.622019:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830603.622023:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830603.622033:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5153 00000100:00080000:3.0:1787830603.622043:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830603.622050:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830603.622085:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281032320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830603.622114:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830603.622118:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830603.622120:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830603.622125:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5153 00010000:00010000:0.0:1787830603.622129:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830603.622129:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830603.622136:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830603.622150:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830603.622162:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281032448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830603.622171:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830603.622174:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830603.622176:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830603.622180:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5153 00000100:00080000:3.0:1787830603.622183:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830603.622187:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830603.622192:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830603.622202:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281032576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830603.622211:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830603.622214:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830603.622215:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830603.622219:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5153 00000100:00080000:3.0:1787830603.622222:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830603.622226:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830603.622230:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830603.622241:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281032704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830603.622250:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830603.622253:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830603.622254:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830603.622258:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5153 00000100:00080000:3.0:1787830603.622261:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830603.622264:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830603.622268:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830603.622278:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281032832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830603.623224:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281032320:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830603.623494:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830603.623549:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830603/real 1787830603] req@ffff8ab3d0a41e00 x1874671281032320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830603.623568:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281032320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830603.623579:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830603.623584:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830603.623588:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281032320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830603.623599:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281032448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830603.623635:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830603.623647:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830603/real 1787830603] req@ffff8ab3d0a430c0 x1874671281032448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830603.623657:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281032448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830603.623663:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830603.623666:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830603.623669:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281032448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830603.623674:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281032576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830603.623689:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830603.623698:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830603/real 1787830603] req@ffff8ab3d0a421c0 x1874671281032576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830603.623714:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281032576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830603.623722:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830603.623724:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830603.623728:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281032576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830603.623734:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281032704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830603.623752:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830603.623762:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830603/real 1787830603] req@ffff8ab3d0a40000 x1874671281032704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830603.623773:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281032704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830603.623782:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830603.623785:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830603.623789:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281032704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830603.623795:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281032832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830603.623820:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830603.623830:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830603/real 1787830603] req@ffff8ab3d0a43c00 x1874671281032832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830603.623840:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281032832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830603.623848:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830603.623851:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830603.623854:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281032832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830604.646132:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830605.670092:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830606.694364:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830607.718987:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830608.742011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830608.742021:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830608.742026:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830608.742036:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5158 00000100:00080000:3.0:1787830608.742047:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830608.742054:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830608.742091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281032960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830608.742119:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830608.742123:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830608.742125:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830608.742129:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5158 00000100:00080000:3.0:1787830608.742133:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830608.742138:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830608.742149:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830608.742155:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281032960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830608.742162:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281033088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830608.742174:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830608.742178:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830608.742180:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830608.742184:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5158 00000100:00080000:3.0:1787830608.742188:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830608.742192:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830608.742197:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830608.742209:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281033216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830608.742222:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830608.742229:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830608.742233:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830608.742235:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830608.742241:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5158 00000100:00080000:3.0:1787830608.742245:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830608.742250:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830608.742255:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830608.742266:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281033344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830608.742274:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830608.742278:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830608.742280:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830608.742286:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5158 00000100:00100000:1.0:1787830608.742287:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830608/real 1787830608] req@ffff8ab3d0a412c0 x1874671281032960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830608.742289:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830608.742298:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830608.742306:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830608.742314:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281032960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830608.742320:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281033472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830608.742324:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830608.742332:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830608.742339:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281032960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830608.742364:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281033088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830608.742387:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830608.742401:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830608/real 1787830608] req@ffff8ab3d0a43840 x1874671281033088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830608.742412:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281033088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830608.742420:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830608.742434:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830608.742437:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281033088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830608.742447:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281033216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830608.742466:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830608.742477:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830608/real 1787830608] req@ffff8ab3d0a43480 x1874671281033216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830608.742488:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281033216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830608.742502:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830608.742504:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830608.742507:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281033216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830608.742513:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281033344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830608.742532:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830608.742542:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830608/real 1787830608] req@ffff8ab3d0a43c00 x1874671281033344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830608.742553:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281033344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830608.742562:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830608.742565:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830608.742568:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281033344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830608.742574:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281033472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830608.742591:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830608.742602:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830608/real 1787830608] req@ffff8ab3d0a40000 x1874671281033472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830608.742611:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281033472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830608.742620:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830608.742623:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830608.742627:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281033472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830608.744312:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830609.766236:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830610.790097:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830611.815066:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830612.838173:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830613.862194:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830613.862205:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830613.862210:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830613.862221:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5163 00010000:00010000:3.0:1787830613.862224:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830613.862231:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830613.862240:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830613.862279:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671281033600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830613.862308:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830613.862312:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830613.862314:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830613.862318:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5163 00000100:00080000:1.0:1787830613.862332:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830613.862337:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830613.862344:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281033600:192.168.203.124@tcp:250:kworker.0 00000008:00100000:1.0:1787830613.862347:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830613.862362:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671281033728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830613.862495:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830613.862499:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830613.862501:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830613.862506:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5163 00000100:00080000:1.0:1787830613.862511:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830613.862515:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830613.862521:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830613.862534:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671281033856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830613.862547:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281033728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830613.862574:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281033856:192.168.203.124@tcp:38:kworker.0 00000100:00080000:1.0:1787830613.862631:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830613.862636:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830613.862638:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830613.862642:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5163 00000100:00080000:1.0:1787830613.862647:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830613.862652:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830613.862657:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830613.862669:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671281033984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830613.862690:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830613.862693:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830613.862695:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830613.862698:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5163 00000100:00080000:1.0:1787830613.862702:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830613.862706:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830613.862710:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830613.862718:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671281034112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830613.862781:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281033984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830613.862812:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281034112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830614.886329:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830615.910212:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787830616.934274:0:10944:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:3.0:1787830616.934315:0:10944:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787830616.934331:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830616.934382:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830616.934396:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830616.934404:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830616.934411:0:10944:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00010000:00010000:2.0:1787830616.934523:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830616.934641:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830613/real 1787830616] req@ffff8ab3d22fe1c0 x1874671281033984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830616.934665:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671281033984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830616.934679:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830616.934684:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830616.934688:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281033984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830616.934696:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830613/real 1787830616] req@ffff8ab3d22fde00 x1874671281034112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830616.934708:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671281034112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830616.934717:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830616.934720:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830616.934723:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281034112:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830616.934729:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830613/real 1787830616] req@ffff8ab3d22fd2c0 x1874671281033856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830616.934739:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671281033856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830616.934747:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830616.934750:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830616.934753:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281033856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830616.934759:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830613/real 1787830616] req@ffff8ab3d22fc780 x1874671281033728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830616.934769:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671281033728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830616.934777:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830616.934780:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830616.934783:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281033728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830616.934789:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830613/real 1787830616] req@ffff8ab3d22fed00 x1874671281033600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830616.934800:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671281033600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830616.934807:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830616.934811:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830616.934814:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281033600:192.168.203.124@tcp:250:kworker.0 00010000:00010000:0.0:1787830617.958209:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830618.982335:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830618.982724:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830618.982730:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830618.982735:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830618.982745:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5168 00000100:00080000:3.0:1787830618.982755:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830618.982761:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830618.982791:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281034240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830618.982818:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830618.982823:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830618.982825:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830618.982828:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5168 00000100:00080000:3.0:1787830618.982832:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830618.982836:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830618.982846:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830618.982859:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281034368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830618.982868:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830618.982872:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830618.982873:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830618.982878:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5168 00000100:00080000:3.0:1787830618.982882:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830618.983017:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281034240:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830618.983086:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830618.983127:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830618/real 1787830618] req@ffff8ab3d0a421c0 x1874671281034240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830618.983143:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281034240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830618.983152:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830618.983156:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830618.983160:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281034240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830618.983167:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281034368:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830618.983198:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830618.983210:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830618/real 1787830618] req@ffff8ab3d0a430c0 x1874671281034368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830618.983222:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281034368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830618.983229:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830618.983232:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830618.983234:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281034368:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787830618.983378:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830618.983385:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830618.983399:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281034496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830618.983417:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830618.983420:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830618.983422:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830618.983425:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5168 00000100:00080000:3.0:1787830618.983429:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830618.983434:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830618.983438:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830618.983449:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281034624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830618.983457:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830618.983460:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830618.983462:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830618.983465:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5168 00000100:00080000:3.0:1787830618.983468:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830618.983471:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830618.983475:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830618.983484:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281034752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830618.983695:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281034496:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830618.983720:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830618.983733:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830618/real 1787830618] req@ffff8ab3d0a41e00 x1874671281034496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830618.983750:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281034496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830618.983758:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830618.983761:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830618.983764:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281034496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830618.983770:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281034624:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830618.983788:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830618.983798:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830618/real 1787830618] req@ffff8ab3d0a430c0 x1874671281034624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830618.983808:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281034624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830618.983816:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830618.983819:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830618.983822:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281034624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830618.983827:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281034752:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830618.983844:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830618.983854:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830618/real 1787830618] req@ffff8ab3d0a421c0 x1874671281034752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830618.983865:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281034752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830618.983880:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830618.983883:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830618.984842:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281034752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830620.006203:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830621.030066:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830622.054166:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830623.079121:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830624.102177:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830624.104945:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830624.104954:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830624.104959:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830624.104969:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5173 00000100:00080000:0.0:1787830624.104979:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830624.104987:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830624.105022:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671281034880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830624.105061:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830624.105065:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830624.105066:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830624.105070:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5173 00000100:00080000:0.0:1787830624.105073:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830624.105077:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830624.105085:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830624.105099:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671281035008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830624.105108:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830624.105111:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830624.105112:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830624.105116:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5173 00000100:00080000:0.0:1787830624.105119:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830624.105122:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830624.105127:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830624.105136:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671281035136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830624.105148:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830624.105152:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830624.105153:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830624.105157:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5173 00000100:00080000:0.0:1787830624.105160:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830624.105164:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830624.105174:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830624.105184:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671281035264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830624.105191:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830624.105194:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830624.105196:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830624.105200:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5173 00000100:00080000:0.0:1787830624.105203:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830624.105207:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830624.105211:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830624.105219:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671281035392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830624.105331:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281034880:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830624.105389:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830624.105433:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830624/real 1787830624] req@ffff8ab3f075cf00 x1874671281034880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830624.105448:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671281034880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830624.105457:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830624.105461:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830624.105464:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281034880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830624.105471:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281035008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830624.105490:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830624.105502:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830624/real 1787830624] req@ffff8ab3f075e940 x1874671281035008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830624.105513:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671281035008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830624.105521:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830624.105524:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830624.105528:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281035008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830624.105534:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281035136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830624.105552:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830624.105562:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830624/real 1787830624] req@ffff8ab3f075de00 x1874671281035136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830624.105573:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671281035136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830624.105580:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830624.105583:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830624.105587:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281035136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830624.105592:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281035264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830624.105610:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830624.105621:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830624/real 1787830624] req@ffff8ab3f075c000 x1874671281035264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830624.105632:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671281035264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830624.105641:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830624.105645:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830624.105648:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281035264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830624.105654:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281035392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830624.105672:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830624.105683:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830624/real 1787830624] req@ffff8ab3f075cb40 x1874671281035392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830624.105695:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671281035392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830624.105703:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830624.105705:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830624.105708:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281035392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830625.126661:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830626.150156:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830627.175443:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830628.200451:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830629.222174:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830629.222184:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830629.222189:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830629.222199:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5178 00000100:00080000:0.0:1787830629.222210:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1787830629.222213:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830629.222220:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830629.222259:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671281035520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830629.222303:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830629.222307:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830629.222308:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830629.222312:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5178 00000100:00080000:0.0:1787830629.222316:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830629.222320:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830629.222335:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830629.222347:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671281035648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830629.222357:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830629.222360:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830629.222361:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830629.222365:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5178 00000100:00080000:0.0:1787830629.222369:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830629.222372:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830629.222376:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830629.222385:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671281035776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830629.222394:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830629.222397:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830629.222399:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830629.222402:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5178 00000100:00080000:0.0:1787830629.222405:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830629.222409:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830629.222414:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830629.222433:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671281035904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830629.222442:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830629.222445:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830629.222446:0:36:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830629.222450:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5178 00000100:00080000:0.0:1787830629.222453:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830629.222457:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830629.222461:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830629.222470:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671281036032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830629.222595:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281035520:192.168.203.124@tcp:250:kworker.0 00000800:00000100:2.0:1787830629.222657:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830629.222702:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830629/real 1787830629] req@ffff8ab3f075f0c0 x1874671281035520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830629.222719:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671281035520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830629.222729:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830629.222734:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830629.222737:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281035520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830629.222744:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281035648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830629.222766:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830629.222779:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830629/real 1787830629] req@ffff8ab3f075f840 x1874671281035648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830629.222790:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671281035648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830629.222797:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830629.222800:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830629.222803:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281035648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830629.222809:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281035776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830629.222826:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830629.222836:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830629/real 1787830629] req@ffff8ab3f075ed00 x1874671281035776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830629.222853:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671281035776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830629.222861:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830629.222864:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830629.222867:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281035776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830629.222872:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281035904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830629.222916:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830629.222927:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830629/real 1787830629] req@ffff8ab3f075f480 x1874671281035904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830629.222938:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671281035904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830629.222951:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830629.222955:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830629.222959:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281035904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830629.222966:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281036032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830629.222984:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830629.224110:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830629/real 1787830629] req@ffff8ab3f075e580 x1874671281036032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830629.224127:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671281036032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830629.224143:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830629.224146:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830629.224150:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281036032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830630.246150:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830631.270119:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830632.294083:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830633.320349:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830634.343093:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830634.343106:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830634.343111:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830634.343124:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5183 00000100:00080000:1.0:1787830634.343136:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830634.343144:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830634.343200:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671281036160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830634.343389:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281036160:192.168.203.124@tcp:250:kworker.0 00000800:00000100:2.0:1787830634.343465:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00010000:00010000:3.0:1787830634.343531:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830634.343816:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830634.343821:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830634.343823:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830634.343828:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5183 00000100:00080000:1.0:1787830634.343832:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830634.343837:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830634.343857:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830634.343874:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671281036288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830634.344497:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830634/real 1787830634] req@ffff8ab3d22fd680 x1874671281036160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830634.344785:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671281036160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830634.344798:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830634.344802:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830634.344807:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281036160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830634.344836:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281036288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830634.344976:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830634.344992:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830634/real 1787830634] req@ffff8ab3d22fcb40 x1874671281036288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830634.345004:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671281036288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830634.345012:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830634.345016:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830634.345019:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281036288:192.168.203.124@tcp:38:kworker.0 00000100:00080000:1.0:1787830634.345133:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830634.345138:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830634.345141:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830634.345147:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5183 00000100:00080000:1.0:1787830634.345152:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830634.345160:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830634.345346:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830634.345370:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671281036416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830634.345419:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281036416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830634.345445:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:1.0:1787830634.345469:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830634.345474:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830634.345476:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830634.345481:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5183 00000100:00080000:1.0:1787830634.345485:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830634.345490:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830634.345496:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830634.345519:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671281036544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830634.345760:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830634/real 1787830634] req@ffff8ab3d22fda40 x1874671281036416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830634.345775:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671281036416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830634.345784:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830634.345787:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830634.345791:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281036416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830634.345805:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281036544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830634.345827:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830634.345839:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830634/real 1787830634] req@ffff8ab3d22fc000 x1874671281036544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830634.345850:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671281036544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830634.345860:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830634.345864:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830634.345867:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281036544:192.168.203.124@tcp:8:kworker.0 00000100:00080000:1.0:1787830634.345993:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830634.345998:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830634.346000:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830634.346004:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5183 00000100:00080000:1.0:1787830634.346009:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830634.346014:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830634.346020:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830634.346033:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671281036672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830634.346078:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281036672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830634.346101:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830634.346291:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830634/real 1787830634] req@ffff8ab3d22ff480 x1874671281036672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830634.346307:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671281036672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830634.346318:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830634.346321:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830634.346324:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281036672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830635.366156:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830636.390988:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830637.414965:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830638.438445:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830639.462337:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830639.463377:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830639.463383:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830639.463387:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830639.463394:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5189 00000100:00080000:3.0:1787830639.463403:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830639.463407:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830639.463438:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281036800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830639.463461:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830639.463466:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830639.463468:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830639.463471:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5189 00000100:00080000:3.0:1787830639.463475:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830639.463479:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830639.463488:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830639.463501:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281036928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830639.463512:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281036800:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787830639.463518:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830639.463522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830639.463523:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830639.463528:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5189 00000100:00080000:3.0:1787830639.463531:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830639.463535:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830639.463539:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830639.463552:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281037056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830639.463560:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830639.463563:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830639.463565:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830639.463568:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5189 00000100:00080000:3.0:1787830639.463571:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830639.463577:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:2.0:1787830639.463579:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787830639.463584:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830639.463596:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281037184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830639.463605:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830639.463608:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830639.463610:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830639.463614:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5189 00000100:00080000:3.0:1787830639.463617:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830639.463621:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830639.463626:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830639.463633:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830639/real 1787830639] req@ffff8ab3d0a40000 x1874671281036800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830639.463638:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281037312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830639.463663:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281036800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830639.463674:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830639.463679:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830639.463682:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281036800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830639.463708:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281036928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830639.463735:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830639.463749:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830639/real 1787830639] req@ffff8ab3d0a43c00 x1874671281036928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830639.463761:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281036928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830639.463775:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830639.463778:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830639.463782:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281036928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830639.463787:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281037056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830639.463805:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830639.463814:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830639/real 1787830639] req@ffff8ab3d0a43480 x1874671281037056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830639.463825:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281037056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830639.463833:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830639.463836:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830639.463839:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281037056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830639.463844:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281037184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830639.463860:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830639.463870:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830639/real 1787830639] req@ffff8ab3d0a43840 x1874671281037184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830639.463881:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281037184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830639.463915:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830639.463919:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830639.463923:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281037184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830639.463937:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281037312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830639.463955:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830639.463966:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830639/real 1787830639] req@ffff8ab3d0a412c0 x1874671281037312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830639.463978:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281037312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830639.463987:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830639.463990:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830639.463993:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281037312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830640.486236:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830641.510187:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830642.536489:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830643.558634:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830644.582562:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830644.582652:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830644.582658:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830644.582663:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830644.582672:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5194 00000100:00080000:1.0:1787830644.582682:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830644.582688:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830644.582725:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22ff480 x1874671281037440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830644.582751:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830644.582754:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830644.582756:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830644.582760:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5194 00000100:00080000:1.0:1787830644.582763:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830644.582767:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830644.582776:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830644.582787:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc000 x1874671281037568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830644.582797:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830644.582803:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830644.582805:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830644.582808:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5194 00000100:00080000:1.0:1787830644.582811:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830644.582815:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830644.582819:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830644.582828:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fda40 x1874671281037696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830644.582838:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830644.582841:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830644.582843:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830644.582846:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5194 00000100:00080000:1.0:1787830644.582849:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830644.582853:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830644.582858:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830644.582866:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fcb40 x1874671281037824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830644.582874:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830644.582877:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830644.582879:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830644.582883:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5194 00000100:00080000:1.0:1787830644.582902:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830644.582906:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830644.582910:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830644.582920:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd680 x1874671281037952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830644.582992:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281037440:192.168.203.124@tcp:250:kworker.0 00000800:00000100:2.0:1787830644.583046:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830644.583379:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830644/real 1787830644] req@ffff8ab3d22ff480 x1874671281037440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830644.583400:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22ff480 x1874671281037440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830644.583411:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830644.583414:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830644.583419:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281037440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830644.583426:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281037568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830644.583447:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830644.583459:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830644/real 1787830644] req@ffff8ab3d22fc000 x1874671281037568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830644.583470:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc000 x1874671281037568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830644.583477:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830644.583480:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830644.583483:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281037568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830644.583489:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281037696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830644.583505:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830644.583514:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830644/real 1787830644] req@ffff8ab3d22fda40 x1874671281037696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830644.583525:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fda40 x1874671281037696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830644.583533:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830644.583536:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830644.583539:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fda40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281037696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830644.583544:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281037824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830644.583560:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830644.583569:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830644/real 1787830644] req@ffff8ab3d22fcb40 x1874671281037824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830644.583579:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fcb40 x1874671281037824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830644.583589:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830644.583592:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830644.583595:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fcb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281037824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830644.583600:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281037952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830644.583616:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830644.583626:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830644/real 1787830644] req@ffff8ab3d22fd680 x1874671281037952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830644.583637:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd680 x1874671281037952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830644.583646:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830644.583649:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830644.583652:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281037952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830645.608384:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830646.630325:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830647.654664:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830648.678203:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830649.701981:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830649.701991:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830649.701996:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830649.702005:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5199 00000100:00080000:1.0:1787830649.702016:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830649.702023:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830649.702061:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fed00 x1874671281038080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830649.702089:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830649.702092:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830649.702093:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830649.702096:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5199 00000100:00080000:1.0:1787830649.702099:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830649.702102:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830649.702110:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830649.702122:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fc780 x1874671281038208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830649.702132:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830649.702135:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830649.702137:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830649.702153:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5199 00000100:00080000:1.0:1787830649.702157:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830649.702161:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830649.702166:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830649.702177:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fd2c0 x1874671281038336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830649.702185:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830649.702188:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830649.702189:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830649.702193:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5199 00000100:00080000:1.0:1787830649.702196:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830649.702200:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830649.702205:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830649.702214:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fde00 x1874671281038464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830649.702222:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830649.702225:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830649.702226:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830649.702229:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5199 00000100:00080000:1.0:1787830649.702232:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830649.702235:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:1.0:1787830649.702238:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830649.702248:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d22fe1c0 x1874671281038592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830649.703432:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281038080:192.168.203.124@tcp:250:kworker.0 00000800:00000100:2.0:1787830649.703498:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830649.703544:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830649/real 1787830649] req@ffff8ab3d22fed00 x1874671281038080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830649.703561:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fed00 x1874671281038080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830649.703571:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830649.703575:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830649.703578:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281038080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830649.703586:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281038208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830649.703645:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830649.703659:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830649/real 1787830649] req@ffff8ab3d22fc780 x1874671281038208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1787830649.703676:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830649.703746:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fc780 x1874671281038208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830649.703756:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830649.703759:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830649.703763:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fc780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281038208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830649.703769:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281038336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830649.703795:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830649.703808:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830649/real 1787830649] req@ffff8ab3d22fd2c0 x1874671281038336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830649.703818:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fd2c0 x1874671281038336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830649.703833:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830649.703836:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830649.703839:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fd2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281038336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830649.703844:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281038464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830649.703858:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830649.703866:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830649/real 1787830649] req@ffff8ab3d22fde00 x1874671281038464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830649.703875:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fde00 x1874671281038464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830649.703883:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830649.703913:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830649.703916:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fde00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281038464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830649.703923:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281038592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830649.703938:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830649.703947:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830649/real 1787830649] req@ffff8ab3d22fe1c0 x1874671281038592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830649.703957:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d22fe1c0 x1874671281038592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830649.703966:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830649.703969:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830649.703973:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d22fe1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281038592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830650.726647:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830651.750917:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830652.774204:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830653.798065:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830654.822161:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830654.822170:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830654.822175:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830654.822184:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5204 00000100:00080000:3.0:1787830654.822195:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830654.822202:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830654.822238:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281038720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830654.822280:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830654.822284:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830654.822285:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830654.822289:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5204 00000100:00080000:3.0:1787830654.822292:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830654.822296:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830654.822306:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830654.822316:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281038848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830654.822325:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830654.822327:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830654.822328:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830654.822332:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5204 00000100:00080000:3.0:1787830654.822335:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830654.822338:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830654.822342:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830654.822350:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281038976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830654.822357:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830654.822360:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830654.822362:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830654.822364:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5204 00000100:00080000:3.0:1787830654.822367:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830654.822371:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830654.822375:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830654.822383:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281039104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830654.822391:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830654.822394:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830654.822396:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830654.822399:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5204 00000100:00080000:3.0:1787830654.822402:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830654.822405:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830654.822409:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830654.822416:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281039232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830654.822570:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281038720:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830654.822629:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830654.822675:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830654/real 1787830654] req@ffff8ab3d0a412c0 x1874671281038720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830654.822690:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281038720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830654.822699:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830654.822702:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830654.822706:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281038720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830654.822713:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281038848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830654.822734:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830654.822745:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830654/real 1787830654] req@ffff8ab3d0a43840 x1874671281038848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830654.822756:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281038848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830654.822763:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830654.822766:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830654.822769:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281038848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830654.822774:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281038976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830654.822793:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830654.822817:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830654/real 1787830654] req@ffff8ab3d0a43480 x1874671281038976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830654.822827:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281038976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830654.822834:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830654.822837:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830654.822840:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281038976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830654.822845:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281039104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830654.822865:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830654.822875:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830654/real 1787830654] req@ffff8ab3d0a43c00 x1874671281039104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830654.822915:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281039104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830654.822927:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830654.822930:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830654.822933:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281039104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830654.822939:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281039232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830654.822958:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830654.822969:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830654/real 1787830654] req@ffff8ab3d0a40000 x1874671281039232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830654.822979:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281039232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830654.822988:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830654.822990:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830654.822993:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281039232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830654.823437:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830655.846319:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830656.870212:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830657.894044:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830658.918161:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830659.942020:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830659.942030:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830659.942034:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830659.942044:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5209 00000100:00080000:3.0:1787830659.942057:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830659.942064:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830659.942113:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281039360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787830659.942129:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830659.942182:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281039360:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830659.942238:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830659.942269:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830659.942273:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830659.942276:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830659.942280:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5209 00000100:00080000:3.0:1787830659.942284:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830659.942288:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830659.942297:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830659.942309:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281039488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830659.943057:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830659/real 1787830659] req@ffff8ab3d0a421c0 x1874671281039360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830659.943086:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281039360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830659.943094:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830659.943097:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830659.943099:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281039360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830659.943120:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281039488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830659.943149:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830659.943162:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830659/real 1787830659] req@ffff8ab3d0a430c0 x1874671281039488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830659.943174:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281039488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830659.943182:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830659.943185:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830659.943188:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281039488:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787830659.945159:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830659.945169:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830659.945173:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830659.945181:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5209 00000100:00080000:3.0:1787830659.945189:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830659.945196:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830659.945208:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830659.945238:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281039616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830659.945257:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830659.945260:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830659.945261:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830659.945266:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5209 00000100:00080000:3.0:1787830659.945269:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830659.945274:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830659.945279:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:1.0:1787830659.945283:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281039616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:3.0:1787830659.945295:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281039744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830659.945306:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830659.945309:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830659.945311:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830659.945315:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5209 00000100:00080000:3.0:1787830659.945329:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830659.945333:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000800:00000100:1.0:1787830659.945339:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000008:00100000:3.0:1787830659.945341:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830659.945355:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281039872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830659.945366:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830659/real 1787830659] req@ffff8ab3d0a41e00 x1874671281039616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830659.945382:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281039616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830659.945391:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830659.945395:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830659.945398:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281039616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830659.945416:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281039744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830659.945433:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830659.945443:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830659/real 1787830659] req@ffff8ab3d0a430c0 x1874671281039744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830659.945453:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281039744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830659.945462:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830659.945464:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830659.945467:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281039744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830659.945472:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281039872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830659.945485:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830659.945494:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830659/real 1787830659] req@ffff8ab3d0a421c0 x1874671281039872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830659.945504:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281039872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830659.945512:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830659.945515:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830659.945517:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281039872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830660.966243:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830661.991547:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830663.014281:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830664.038277:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830665.064228:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830665.064239:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830665.064243:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830665.064254:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5214 00000100:00080000:0.0:1787830665.064265:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830665.064274:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830665.064311:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671281040000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830665.064344:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830665.064348:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830665.064350:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830665.064354:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5214 00000100:00080000:0.0:1787830665.064357:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830665.064362:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830665.064372:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830665.064386:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671281040128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830665.064396:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830665.064399:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830665.064401:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830665.064404:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5214 00000100:00080000:0.0:1787830665.064408:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830665.064411:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830665.064416:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830665.064425:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671281040256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830665.064434:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830665.064437:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830665.064438:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830665.064442:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5214 00000100:00080000:0.0:1787830665.064446:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830665.064450:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830665.064455:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830665.064465:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671281040384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830665.064473:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830665.064476:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830665.064478:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830665.064481:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5214 00000100:00080000:0.0:1787830665.064484:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830665.064488:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830665.064492:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830665.064501:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671281040512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830665.064590:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281040000:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830665.064647:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830665.064691:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830665/real 1787830665] req@ffff8ab3f075da40 x1874671281040000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830665.064707:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671281040000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830665.064717:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830665.064720:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830665.064724:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281040000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830665.064731:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281040128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830665.064753:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830665.064766:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830665/real 1787830665] req@ffff8ab3f075e580 x1874671281040128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830665.064777:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671281040128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830665.064790:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830665.064793:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830665.064796:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281040128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830665.064801:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281040256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830665.064818:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830665.064827:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830665/real 1787830665] req@ffff8ab3f075f480 x1874671281040256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830665.064837:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671281040256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830665.064844:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830665.064847:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830665.064850:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281040256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830665.064855:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281040384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830665.064871:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830665.064879:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830665/real 1787830665] req@ffff8ab3f075ed00 x1874671281040384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830665.064911:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671281040384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830665.064921:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830665.064924:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830665.064928:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281040384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830665.064937:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281040512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830665.064955:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830665.064966:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830665/real 1787830665] req@ffff8ab3f075f840 x1874671281040512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830665.064976:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671281040512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830665.064985:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830665.064988:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830665.064991:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281040512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830665.065086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830666.086116:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830667.110302:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830668.134402:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830669.158167:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830670.182117:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1787830670.182121:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830670.182128:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830670.182132:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830670.182142:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5219 00000100:00080000:2.0:1787830670.182225:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830670.182233:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830670.182271:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671281040640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830670.182373:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281040640:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830670.182432:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830670.182464:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830670.182468:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830670.182469:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830670.182473:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5219 00000100:00080000:2.0:1787830670.182477:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830670.182481:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830670.182490:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830670.182504:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671281040768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830670.184693:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830670/real 1787830670] req@ffff8ab3f0439680 x1874671281040640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830670.184719:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671281040640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830670.184730:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830670.184734:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830670.184738:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281040640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830670.184767:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281040768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830670.184820:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830670.184836:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830670/real 1787830670] req@ffff8ab3f04392c0 x1874671281040768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830670.184847:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671281040768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830670.184855:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830670.184859:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830670.184862:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281040768:192.168.203.124@tcp:38:kworker.0 00000100:00080000:2.0:1787830670.185750:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830670.185758:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830670.185761:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830670.185768:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5219 00000100:00080000:2.0:1787830670.185775:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830670.185782:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830670.185801:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830670.185830:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671281040896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830670.185848:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830670.185852:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830670.185853:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830670.185857:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5219 00000100:00080000:2.0:1787830670.185861:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830670.185865:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830670.185871:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830670.185882:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671281041024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830670.185914:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830670.185917:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830670.185919:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830670.185923:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5219 00000100:00080000:2.0:1787830670.185925:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830670.185929:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830670.185934:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830670.185945:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671281041152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830670.186272:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281040896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830670.186315:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830670.186339:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830670/real 1787830670] req@ffff8ab3f043ad00 x1874671281040896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830670.186352:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671281040896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830670.186361:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830670.186365:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830670.186369:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281040896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830670.186375:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281041024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830670.186394:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830670.186404:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830670/real 1787830670] req@ffff8ab3f04383c0 x1874671281041024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830670.186420:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671281041024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830670.186429:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830670.186432:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830670.186435:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281041024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830670.186440:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281041152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830670.186456:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830670.186466:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830670/real 1787830670] req@ffff8ab3f043bc00 x1874671281041152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830670.186476:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671281041152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830670.186485:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830670.186488:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830670.186491:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281041152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830671.206180:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830672.230249:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830673.254049:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830674.278202:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830675.302507:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830675.302519:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830675.302706:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830675.302717:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5224 00000100:00080000:3.0:1787830675.302728:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830675.302734:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830675.302771:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281041280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830675.302838:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281041280:192.168.203.124@tcp:250:kworker.0 00000800:00000100:2.0:1787830675.302912:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830675.302977:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830675.302981:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830675.302983:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830675.302988:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5224 00000100:00080000:3.0:1787830675.302992:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830675.302996:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830675.303006:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830675.303019:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281041408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787830675.303123:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830675.303207:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830675/real 1787830675] req@ffff8ab3d0a40000 x1874671281041280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830675.303224:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281041280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830675.303233:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830675.303236:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830675.303239:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281041280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830675.303262:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281041408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830675.303284:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830675.303297:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830675/real 1787830675] req@ffff8ab3d0a43c00 x1874671281041408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830675.303308:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281041408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830675.303315:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830675.303318:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830675.303321:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281041408:192.168.203.124@tcp:38:kworker.0 00000100:00080000:3.0:1787830675.303600:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830675.303605:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830675.303607:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830675.303612:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5224 00000100:00080000:3.0:1787830675.303617:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830675.303621:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830675.303628:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830675.303642:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281041536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830675.303778:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281041536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830675.303803:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830675.303826:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830675.303829:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830675.303831:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830675.303836:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5224 00000100:00080000:3.0:1787830675.303840:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830675.303844:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830675.303850:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830675.303863:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281041664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830675.303924:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830675/real 1787830675] req@ffff8ab3d0a43480 x1874671281041536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830675.303937:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281041536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830675.303945:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830675.303949:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830675.303952:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281041536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830675.303974:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281041664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830675.303997:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830675.304009:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830675/real 1787830675] req@ffff8ab3d0a43840 x1874671281041664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830675.304021:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281041664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830675.304036:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830675.304039:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830675.304042:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281041664:192.168.203.124@tcp:8:kworker.0 00000100:00080000:3.0:1787830675.304294:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830675.304299:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830675.304301:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830675.304306:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5224 00000100:00080000:3.0:1787830675.304310:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830675.304315:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830675.304320:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830675.304335:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281041792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830675.304402:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281041792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830675.304427:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830675.304441:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830675/real 1787830675] req@ffff8ab3d0a412c0 x1874671281041792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830675.304451:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281041792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830675.304459:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830675.304462:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830675.304465:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281041792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830676.329060:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830677.352128:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830678.378325:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830679.398064:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830680.422128:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830680.422423:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830680.422430:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830680.422433:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830680.422443:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5230 00000100:00080000:3.0:1787830680.422453:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830680.422460:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830680.422498:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281041920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830680.422524:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830680.422528:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830680.422529:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830680.422532:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5230 00000100:00080000:3.0:1787830680.422535:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830680.422539:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830680.422549:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830680.422560:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281042048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830680.422569:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830680.422572:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830680.422574:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830680.422577:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5230 00000100:00080000:3.0:1787830680.422581:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830680.422584:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830680.422589:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830680.422599:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281042176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830680.422607:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830680.422610:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830680.422612:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830680.422616:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5230 00000100:00080000:3.0:1787830680.422619:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830680.422624:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830680.422629:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830680.422637:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281041920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830680.422641:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281042304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830680.422656:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830680.422660:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830680.422661:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830680.422665:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5230 00000100:00080000:3.0:1787830680.422669:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830680.422673:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830680.422678:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830680.422687:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281042432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830680.422739:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281042048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830680.422771:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281042176:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830680.422802:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281042304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830680.422827:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281042432:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830681.446177:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830682.470278:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830683.494062:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787830683.494608:0:10943:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:2.0:1787830683.494649:0:10943:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787830683.494665:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787830683.494696:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787830683.494705:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787830683.494718:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:2.0:1787830683.494727:0:10943:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:2.0:1787830683.494757:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830680/real 1787830683] req@ffff8ab3d0a43c00 x1874671281042304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830683.494779:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281042304/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830683.494791:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830683.494796:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830683.494800:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281042304:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830683.494807:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830680/real 1787830683] req@ffff8ab3d0a40000 x1874671281042432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830683.494821:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281042432/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830683.494830:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830683.494834:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830683.494838:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281042432:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830683.494847:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830680/real 1787830683] req@ffff8ab3d0a412c0 x1874671281041920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830683.494861:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281041920/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830683.494869:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830683.494872:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830683.494875:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281041920:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830683.494881:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830680/real 1787830683] req@ffff8ab3d0a43840 x1874671281042048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830683.494937:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281042048/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830683.494945:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830683.494948:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830683.494952:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281042048:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830683.494958:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830680/real 1787830683] req@ffff8ab3d0a43480 x1874671281042176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830683.494971:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281042176/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830683.494980:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830683.494984:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830683.494988:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281042176:192.168.203.124@tcp:38:kworker.0 00010000:00010000:2.0:1787830684.518173:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830685.542076:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830685.542124:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830685.542128:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830685.542132:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830685.542139:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5235 00000100:00080000:3.0:1787830685.542147:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830685.542152:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830685.542190:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281042560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830685.542218:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830685.542221:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830685.542223:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830685.542227:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5235 00000100:00080000:3.0:1787830685.542230:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830685.542235:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830685.542244:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830685.542254:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281042688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830685.542261:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830685.542264:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830685.542266:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830685.542269:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5235 00000100:00080000:3.0:1787830685.542273:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830685.542276:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830685.542281:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830685.542292:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281042816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830685.542300:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830685.542303:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830685.542305:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830685.542308:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5235 00000100:00080000:3.0:1787830685.542311:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830685.542316:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830685.542319:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830685.542339:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281042560:192.168.203.124@tcp:250:kworker.0 00000800:00000100:2.0:1787830685.542415:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:3.0:1787830685.542458:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281042944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830685.542460:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830685/real 1787830685] req@ffff8ab3d0a421c0 x1874671281042560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830685.542473:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830685.542477:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830685.542479:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830685.542482:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5235 00000100:00080000:3.0:1787830685.542486:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830685.542490:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281042560/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830685.542490:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830685.542498:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:2.0:1787830685.542509:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787830685.542510:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281043072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830685.542518:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830685.542525:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281042560:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830685.542533:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281042688:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830685.542555:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830685.542568:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830685/real 1787830685] req@ffff8ab3d0a430c0 x1874671281042688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830685.542579:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281042688/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830685.542587:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830685.542590:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830685.542594:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281042688:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830685.542600:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281042816:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830685.542617:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830685.542627:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830685/real 1787830685] req@ffff8ab3d0a41e00 x1874671281042816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830685.542637:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281042816/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830685.542644:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830685.542647:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830685.542650:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281042816:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830685.542673:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281042944:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830685.542692:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830685.542702:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830685/real 1787830685] req@ffff8ab3d0a43480 x1874671281042944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830685.542714:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281042944/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830685.542724:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830685.542727:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830685.542730:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281042944:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830685.542735:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281043072:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830685.542750:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830685.542760:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830685/real 1787830685] req@ffff8ab3d0a43840 x1874671281043072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830685.542769:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281043072/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830685.542778:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830685.542780:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830685.542783:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281043072:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830686.566564:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830687.590165:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830688.614240:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830689.638132:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830690.662030:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830690.662039:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830690.662043:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830690.662053:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5240 00000100:00080000:3.0:1787830690.662064:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830690.662072:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830690.662106:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281043200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830690.662132:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830690.662136:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830690.662138:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830690.662142:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5240 00000100:00080000:3.0:1787830690.662145:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830690.662149:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830690.662158:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830690.662171:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281043328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830690.662181:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830690.662186:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830690.662188:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830690.662191:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5240 00000100:00080000:3.0:1787830690.662195:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830690.662198:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830690.662203:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830690.662215:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281043456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830690.662224:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830690.662227:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830690.662229:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830690.662233:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5240 00000100:00080000:3.0:1787830690.662237:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830690.662242:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830690.662247:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830690.662257:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281043584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830690.662266:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830690.662270:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830690.662273:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830690.662274:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281043200:192.168.203.124@tcp:250:kworker.0 00000100:00080000:3.0:1787830690.662280:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5240 00000100:00080000:3.0:1787830690.662283:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830690.662287:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830690.662292:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830690.662306:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281043712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830690.662350:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830690.662396:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830690/real 1787830690] req@ffff8ab3d0a412c0 x1874671281043200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830690.662415:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281043200/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830690.662425:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830690.662430:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830690.662434:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281043200:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830690.662443:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281043328:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830690.662464:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830690.662490:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830690/real 1787830690] req@ffff8ab3d0a40000 x1874671281043328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830690.662503:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281043328/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830690.662510:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830690.662523:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830690.662528:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281043328:192.168.203.124@tcp:38:kworker.0 00010000:00010000:2.0:1787830690.662530:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830690.662538:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281043456:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830690.662559:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830690.662571:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830690/real 1787830690] req@ffff8ab3d0a43c00 x1874671281043456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830690.662583:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281043456/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830690.662591:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830690.662594:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830690.662597:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281043456:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830690.662621:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281043584:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830690.662641:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830690.662653:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830690/real 1787830690] req@ffff8ab3d0a43840 x1874671281043584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830690.662664:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281043584/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830690.662672:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830690.662675:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830690.662678:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281043584:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830690.662683:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281043712:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830690.662700:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830690.662711:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830690/real 1787830690] req@ffff8ab3d0a43480 x1874671281043712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830690.662721:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281043712/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830690.662735:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830690.662738:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830690.662741:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281043712:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830691.686075:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830692.710172:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830693.734053:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830694.758184:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830695.782065:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830695.782075:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830695.782079:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830695.782089:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5245 00000100:00080000:3.0:1787830695.782100:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830695.782107:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830695.782144:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281043840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830695.782170:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830695.782174:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830695.782175:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830695.782179:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5245 00000100:00080000:3.0:1787830695.782182:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830695.782186:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830695.782195:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830695.782206:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281043968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830695.782215:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830695.782218:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830695.782220:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830695.782223:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5245 00000100:00080000:3.0:1787830695.782227:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830695.782231:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830695.782235:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830695.782246:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281044096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830695.782254:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830695.782256:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830695.782258:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830695.782261:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5245 00000100:00080000:3.0:1787830695.782264:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830695.782268:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830695.782273:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830695.782284:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281044224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830695.782292:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830695.782296:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830695.782297:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830695.782301:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5245 00000100:00080000:3.0:1787830695.782304:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830695.782308:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830695.782313:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830695.782322:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281044352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830695.782415:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281043840:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830695.782459:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830695.782500:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830695/real 1787830695] req@ffff8ab3d0a41e00 x1874671281043840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830695.782512:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281043840/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830695.782521:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830695.782525:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830695.782530:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281043840:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830695.782538:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281043968:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830695.782558:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830695.782570:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830695/real 1787830695] req@ffff8ab3d0a430c0 x1874671281043968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830695.782581:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281043968/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830695.782589:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830695.782593:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830695.782596:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281043968:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830695.782602:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281044096:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830695.782619:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830695.782630:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830695/real 1787830695] req@ffff8ab3d0a421c0 x1874671281044096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830695.782642:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281044096/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830695.782650:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830695.782653:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830695.782656:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281044096:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830695.782667:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281044224:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830695.782685:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830695.782696:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830695/real 1787830695] req@ffff8ab3d0a43480 x1874671281044224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830695.782706:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281044224/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830695.782715:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830695.782718:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830695.782721:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281044224:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830695.782727:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281044352:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830695.782744:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830695.782755:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830695/real 1787830695] req@ffff8ab3d0a43840 x1874671281044352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830695.782765:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281044352/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830695.782775:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830695.782778:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830695.782781:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281044352:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830695.782854:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830696.806195:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830697.830151:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830698.854085:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830699.878626:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830700.902353:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830700.902363:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830700.902367:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830700.902385:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5250 00000100:00080000:3.0:1787830700.902395:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830700.902403:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830700.902544:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281044480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830700.902580:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830700.902584:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830700.902586:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830700.902590:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5250 00000100:00080000:3.0:1787830700.902594:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830700.902599:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830700.902608:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830700.902620:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281044608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830700.902630:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830700.902633:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830700.902635:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830700.902639:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5250 00000100:00080000:3.0:1787830700.902642:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830700.902645:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830700.902650:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830700.902659:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281044736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830700.902668:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830700.902672:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830700.902673:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830700.902677:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5250 00000100:00080000:3.0:1787830700.902680:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830700.902684:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830700.902689:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830700.902701:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281044864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830700.902710:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830700.902713:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830700.902715:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830700.902719:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5250 00000100:00080000:3.0:1787830700.902722:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830700.902726:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830700.902731:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830700.902740:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281044992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830700.903287:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281044480:192.168.203.124@tcp:250:kworker.0 00000800:00000100:0.0:1787830700.903344:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830700.903388:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830700/real 1787830700] req@ffff8ab3d0a43c00 x1874671281044480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830700.903569:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281044480/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830700.903579:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830700.903583:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830700.903587:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281044480:192.168.203.124@tcp:250:kworker.0 00000100:00100000:0.0:1787830700.903594:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281044608:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830700.903617:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830700.903631:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830700/real 1787830700] req@ffff8ab3d0a40000 x1874671281044608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830700.903641:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281044608/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830700.903648:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830700.903651:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830700.903654:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281044608:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830700.903660:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281044736:192.168.203.124@tcp:38:kworker.0 00000800:00000100:0.0:1787830700.903677:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830700.903688:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830700/real 1787830700] req@ffff8ab3d0a412c0 x1874671281044736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830700.903698:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281044736/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830700.903705:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830700.903709:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830700.903712:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281044736:192.168.203.124@tcp:38:kworker.0 00000100:00100000:0.0:1787830700.903718:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281044864:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830700.903735:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830700.903745:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830700/real 1787830700] req@ffff8ab3d0a43840 x1874671281044864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830700.903755:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281044864/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830700.903764:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830700.903767:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830700.903771:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281044864:192.168.203.124@tcp:8:kworker.0 00000100:00100000:0.0:1787830700.903776:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281044992:192.168.203.124@tcp:8:kworker.0 00000800:00000100:0.0:1787830700.903808:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830700.903819:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830700/real 1787830700] req@ffff8ab3d0a43480 x1874671281044992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830700.903830:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281044992/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830700.903839:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830700.903842:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830700.903845:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281044992:192.168.203.124@tcp:8:kworker.0 00010000:00010000:2.0:1787830700.904167:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830701.926009:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787830702.950022:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830703.974060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830704.999073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830706.022067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830706.022213:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830706.022219:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830706.022223:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830706.022232:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5255 00000100:00080000:0.0:1787830706.022250:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830706.022257:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830706.022370:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671281045120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830706.022433:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281045120:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830706.022488:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830706.022514:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830706.022516:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830706.022518:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830706.022521:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5255 00000100:00080000:0.0:1787830706.022523:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830706.022526:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830706.022533:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830706.022543:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671281045248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830706.022609:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830706/real 1787830706] req@ffff8ab3f075f0c0 x1874671281045120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830706.022621:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671281045120/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830706.022627:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830706.022630:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830706.022632:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281045120:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830706.022650:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281045248:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830706.022665:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830706.022673:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830706/real 1787830706] req@ffff8ab3f075cb40 x1874671281045248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830706.022680:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671281045248/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830706.022685:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830706.022686:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830706.022688:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281045248:192.168.203.124@tcp:38:kworker.0 00000100:00080000:0.0:1787830706.022721:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830706.022724:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830706.022726:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830706.022729:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5255 00000100:00080000:0.0:1787830706.022732:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830706.022735:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830706.022739:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830706.022760:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671281045376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830706.022774:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830706.022777:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830706.022779:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830706.022783:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5255 00000100:00080000:0.0:1787830706.022787:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830706.022791:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:1.0:1787830706.022792:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281045376:192.168.203.124@tcp:38:kworker.0 00000008:00100000:0.0:1787830706.022795:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830706.022805:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671281045504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830706.022812:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830706.022814:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830706.022817:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830706.022819:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830706.022822:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5255 00000100:00100000:1.0:1787830706.022822:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830706/real 1787830706] req@ffff8ab3f075c000 x1874671281045376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830706.022825:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830706.022829:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830706.022833:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830706.022833:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671281045376/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830706.022841:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787830706.022842:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671281045632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830706.022844:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830706.022847:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281045376:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830706.022858:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281045504:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830706.022876:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830706.022907:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830706/real 1787830706] req@ffff8ab3f075de00 x1874671281045504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830706.022921:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671281045504/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830706.022931:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830706.022935:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830706.022938:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281045504:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830706.022943:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281045632:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830706.022967:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830706.022978:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830706/real 1787830706] req@ffff8ab3f075e940 x1874671281045632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830706.022988:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671281045632/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830706.022997:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830706.023000:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830706.023003:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281045632:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830707.046388:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830708.070067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830709.094164:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830710.118364:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830711.142366:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830711.142377:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830711.142382:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830711.142391:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5260 00000100:00080000:0.0:1787830711.142401:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830711.142407:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830711.142444:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671281045760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830711.142484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830711.142490:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830711.142492:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830711.142495:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5260 00000100:00080000:0.0:1787830711.142499:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830711.142504:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830711.142512:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830711.142527:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671281045888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830711.142536:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830711.142539:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830711.142541:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830711.142544:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5260 00000100:00080000:0.0:1787830711.142547:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830711.142551:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830711.142555:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830711.142565:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671281046016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830711.142574:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830711.142577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830711.142578:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830711.142581:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5260 00000100:00080000:0.0:1787830711.142585:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830711.142589:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830711.142594:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830711.142604:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671281046144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830711.142617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830711.142620:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830711.142622:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830711.142626:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5260 00000100:00080000:0.0:1787830711.142629:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830711.142633:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830711.142637:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830711.142646:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671281046272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830711.142723:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281045760:192.168.203.124@tcp:250:kworker.0 00000800:00000100:2.0:1787830711.142807:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830711.142851:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830711/real 1787830711] req@ffff8ab3f075cf00 x1874671281045760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830711.142868:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671281045760/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830711.142878:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830711.142882:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830711.142902:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281045760:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830711.142910:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281045888:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830711.142930:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830711.142943:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830711/real 1787830711] req@ffff8ab3f075e940 x1874671281045888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830711.142954:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671281045888/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830711.142961:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830711.142964:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830711.142967:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281045888:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830711.142973:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281046016:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830711.142990:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830711.142999:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830711/real 1787830711] req@ffff8ab3f075de00 x1874671281046016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830711.143010:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671281046016/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830711.143018:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830711.143021:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830711.143023:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281046016:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830711.143028:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281046144:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830711.143046:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830711.143056:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830711/real 1787830711] req@ffff8ab3f075c000 x1874671281046144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830711.143066:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671281046144/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830711.143075:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830711.143078:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830711.143081:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281046144:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830711.143087:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281046272:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830711.143102:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830711.143111:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830711/real 1787830711] req@ffff8ab3f075cb40 x1874671281046272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830711.143120:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671281046272/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830711.143128:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830711.143131:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830711.143133:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281046272:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830711.145489:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830712.166149:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830713.190485:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830714.214034:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787830715.238220:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830716.262090:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830716.262095:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830716.262104:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830716.262110:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830716.262121:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5265 00000100:00080000:3.0:1787830716.262131:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830716.262138:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830716.262173:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281046400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830716.262312:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830716.262316:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830716.262318:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830716.262322:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5265 00000100:00080000:3.0:1787830716.262326:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830716.262331:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830716.262340:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830716.262353:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281046528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830716.262363:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830716.262366:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830716.262367:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830716.262371:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5265 00000100:00080000:3.0:1787830716.262375:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830716.262379:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830716.262384:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830716.262394:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281046656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830716.262404:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830716.262407:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830716.262409:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830716.262413:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5265 00000100:00080000:3.0:1787830716.262417:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830716.262422:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830716.262428:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830716.262432:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281046400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830716.262443:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281046784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830716.262454:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830716.262457:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830716.262459:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830716.262462:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5265 00000100:00080000:3.0:1787830716.262466:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830716.262471:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830716.262475:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830716.262487:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281046912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830716.262507:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830716.262566:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830716/real 1787830716] req@ffff8ab3d0a421c0 x1874671281046400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830716.262584:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281046400/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830716.262593:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830716.262598:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830716.262602:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281046400:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830716.262609:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281046528:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830716.262630:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830716.262642:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830716/real 1787830716] req@ffff8ab3d0a430c0 x1874671281046528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830716.262653:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281046528/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830716.262662:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830716.262665:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830716.262668:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281046528:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830716.262686:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281046656:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830716.262706:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830716.262717:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830716/real 1787830716] req@ffff8ab3d0a41e00 x1874671281046656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830716.262728:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281046656/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830716.262736:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830716.262739:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830716.262742:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281046656:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830716.262765:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281046784:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830716.262784:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830716.262795:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830716/real 1787830716] req@ffff8ab3d0a43480 x1874671281046784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830716.262806:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281046784/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830716.262815:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830716.262818:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830716.262821:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281046784:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830716.262826:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281046912:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830716.262843:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830716.262853:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830716/real 1787830716] req@ffff8ab3d0a43840 x1874671281046912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830716.262863:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281046912/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830716.262872:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830716.262875:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830716.262878:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281046912:192.168.203.124@tcp:8:kworker.0 00010000:00010000:3.0:1787830717.286637:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830718.310652:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830719.334294:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830720.358060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830721.382282:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830721.382292:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830721.382297:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830721.382306:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5271 00000100:00080000:3.0:1787830721.382316:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830721.382330:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830721.382368:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281047040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830721.382400:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830721.382403:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830721.382405:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830721.382409:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5271 00000100:00080000:3.0:1787830721.382412:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830721.382416:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830721.382426:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830721.382438:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281047168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830721.382448:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830721.382451:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830721.382452:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830721.382456:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5271 00000100:00080000:3.0:1787830721.382459:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830721.382463:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830721.382467:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830721.382477:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281047296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830721.382486:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830721.382489:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830721.382491:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830721.382494:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5271 00000100:00080000:3.0:1787830721.382498:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830721.382502:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830721.382506:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830721.382517:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281047424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830721.382526:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830721.382528:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830721.382530:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830721.382534:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5271 00000100:00080000:3.0:1787830721.382537:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830721.382540:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830721.382543:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830721.382551:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281047552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830721.382744:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281047040:192.168.203.124@tcp:250:kworker.0 00000800:00000100:2.0:1787830721.382812:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830721.382861:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830721/real 1787830721] req@ffff8ab3d0a412c0 x1874671281047040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830721.382879:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281047040/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1787830721.382898:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830721.382926:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830721.382930:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830721.382934:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281047040:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830721.382941:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281047168:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830721.382962:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830721.382975:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830721/real 1787830721] req@ffff8ab3d0a40000 x1874671281047168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830721.383118:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281047168/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830721.383126:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830721.383129:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830721.383133:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281047168:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830721.383139:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281047296:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830721.383162:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830721.383174:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830721/real 1787830721] req@ffff8ab3d0a43c00 x1874671281047296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830721.383185:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281047296/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830721.383193:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830721.383196:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830721.383200:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281047296:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830721.383206:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281047424:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830721.383225:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830721.383236:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830721/real 1787830721] req@ffff8ab3d0a43840 x1874671281047424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830721.383248:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281047424/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830721.383257:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830721.383260:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830721.383263:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281047424:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830721.383269:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281047552:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830721.383287:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830721.383297:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830721/real 1787830721] req@ffff8ab3d0a43480 x1874671281047552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830721.383308:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281047552/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830721.383317:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830721.383320:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830721.383323:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281047552:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830722.406285:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830723.430163:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830724.454068:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830725.478186:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830726.502614:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830726.502625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830726.502630:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830726.502641:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5276 00000100:00080000:3.0:1787830726.502655:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830726.502663:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830726.503079:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281047680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830726.503106:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830726.503110:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830726.503112:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830726.503116:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5276 00000100:00080000:3.0:1787830726.503120:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830726.503125:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830726.503137:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830726.503150:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281047808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830726.503159:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830726.503163:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830726.503164:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830726.503168:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5276 00000100:00080000:3.0:1787830726.503171:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830726.503175:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830726.503180:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830726.503190:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281047936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830726.503201:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830726.503204:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830726.503205:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830726.503209:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5276 00000100:00080000:3.0:1787830726.503212:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830726.503215:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830726.503220:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830726.503231:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281048064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830726.503240:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830726.503243:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830726.503245:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830726.503249:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5276 00000100:00080000:3.0:1787830726.503252:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830726.503256:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830726.503260:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830726.503269:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281048192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787830726.504990:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830726.505046:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281047680:192.168.203.124@tcp:250:kworker.0 00000800:00000100:2.0:1787830726.505135:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830726.505182:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830726/real 1787830726] req@ffff8ab3d0a41e00 x1874671281047680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830726.505201:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281047680/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830726.505212:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830726.505217:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830726.505220:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281047680:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830726.505228:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281047808:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830726.505248:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830726.505259:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830726/real 1787830726] req@ffff8ab3d0a430c0 x1874671281047808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830726.505269:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281047808/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830726.505745:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830726.505750:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830726.505754:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281047808:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830726.505763:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281047936:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830726.505790:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830726.505803:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830726/real 1787830726] req@ffff8ab3d0a421c0 x1874671281047936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830726.505815:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281047936/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830726.505829:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830726.505832:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830726.505835:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281047936:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830726.505842:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281048064:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830726.505859:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830726.505868:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830726/real 1787830726] req@ffff8ab3d0a43480 x1874671281048064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830726.505878:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281048064/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830726.505914:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830726.505918:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830726.505921:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281048064:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830726.505927:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281048192:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830726.505944:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830726.505953:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830726/real 1787830726] req@ffff8ab3d0a43840 x1874671281048192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830726.505964:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281048192/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830726.505973:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830726.505977:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830726.505980:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281048192:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830727.526079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830728.550083:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830729.574463:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830730.598249:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830731.622120:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830731.622130:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830731.622135:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830731.622146:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5281 00000100:00080000:2.0:1787830731.622156:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830731.622163:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00010000:00010000:1.0:1787830731.622195:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830731.622200:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671281048320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830731.622231:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830731.622235:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830731.622237:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830731.622241:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5281 00000100:00080000:2.0:1787830731.622244:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830731.622257:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830731.622266:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830731.622280:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671281048448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830731.622284:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281048320:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787830731.622299:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830731.622302:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830731.622304:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830731.622309:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5281 00000100:00080000:2.0:1787830731.622312:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830731.622317:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830731.622322:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830731.622333:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671281048576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830731.622342:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830731.622345:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830731.622348:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787830731.622350:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830731.622355:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5281 00000100:00080000:2.0:1787830731.622361:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830731.622366:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830731.622371:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830731.622382:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671281048704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830731.622391:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830731.622394:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830731.622396:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830731.622400:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5281 00000100:00080000:2.0:1787830731.622403:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830731.622408:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830731.622413:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830731.622417:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830731/real 1787830731] req@ffff8ab3f043a1c0 x1874671281048320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830731.622426:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671281048832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830731.622442:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671281048320/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830731.622457:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830731.622461:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830731.622465:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281048320:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830731.622487:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281048448:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830731.622508:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830731.622521:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830731/real 1787830731] req@ffff8ab3f0439e00 x1874671281048448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830731.622532:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671281048448/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830731.622541:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830731.622544:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830731.622548:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281048448:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830731.622553:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281048576:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830731.622584:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830731.622596:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830731/real 1787830731] req@ffff8ab3f043b840 x1874671281048576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830731.622607:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671281048576/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830731.622619:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830731.622622:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830731.622625:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281048576:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830731.622681:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281048704:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830731.622711:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830731.622721:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830731/real 1787830731] req@ffff8ab3f043a940 x1874671281048704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830731.622731:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671281048704/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830731.622739:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830731.622742:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830731.622745:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281048704:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830731.622750:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281048832:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830731.622763:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830731.622770:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830731/real 1787830731] req@ffff8ab3f0439a40 x1874671281048832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830731.622779:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671281048832/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830731.622786:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830731.622789:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830731.622791:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281048832:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830732.646405:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830733.672790:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830734.694634:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830735.719567:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830736.742210:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830736.742341:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830736.742348:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830736.742352:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830736.742362:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5286 00000100:00080000:3.0:1787830736.742373:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830736.742379:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830736.742414:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281048960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830736.742438:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830736.742442:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830736.742444:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830736.742447:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5286 00000100:00080000:3.0:1787830736.742450:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830736.742453:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830736.742462:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830736.742472:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281049088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830736.742481:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830736.742483:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830736.742485:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830736.742488:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5286 00000100:00080000:3.0:1787830736.742490:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830736.742493:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830736.742497:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830736.742504:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281049216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830736.742512:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830736.742516:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830736.742518:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830736.742521:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5286 00000100:00080000:3.0:1787830736.742523:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830736.742527:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830736.742530:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830736.742550:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281049344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830736.742558:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830736.742561:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830736.742562:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830736.742565:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5286 00000100:00080000:3.0:1787830736.742568:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830736.742571:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830736.742575:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830736.742583:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281049472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830736.742649:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281048960:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830736.742711:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830736.742818:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830736/real 1787830736] req@ffff8ab3d0a43c00 x1874671281048960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830736.742836:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281048960/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830736.742846:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830736.742850:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830736.742853:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281048960:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830736.742861:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281049088:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830736.743034:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830736.743048:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830736/real 1787830736] req@ffff8ab3d0a40000 x1874671281049088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830736.745270:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281049088/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830736.745286:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830736.745290:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830736.745294:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281049088:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830736.745306:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281049216:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830736.745357:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830736.745380:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830736/real 1787830736] req@ffff8ab3d0a412c0 x1874671281049216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830736.745393:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281049216/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830736.745400:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830736.745404:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830736.745407:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281049216:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830736.745413:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281049344:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830736.745431:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830736.745440:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830736/real 1787830736] req@ffff8ab3d0a43840 x1874671281049344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830736.745451:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281049344/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830736.745460:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830736.745463:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830736.745466:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281049344:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830736.745472:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281049472:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830736.745486:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830736.745494:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830736/real 1787830736] req@ffff8ab3d0a43480 x1874671281049472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830736.745504:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281049472/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830736.745513:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830736.745515:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830736.745518:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281049472:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830737.766214:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830738.790076:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830739.814057:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830740.838098:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830741.862031:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830741.862039:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830741.862043:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830741.862051:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5291 00000100:00080000:0.0:1787830741.862059:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830741.862065:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830741.862096:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671281049600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787830741.862106:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830741.862119:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830741.862124:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830741.862126:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830741.862130:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5291 00000100:00080000:0.0:1787830741.862134:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830741.862137:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830741.862147:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830741.862158:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671281049728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830741.862167:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830741.862168:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281049600:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830741.862170:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830741.862172:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830741.862175:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5291 00000100:00080000:0.0:1787830741.862178:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830741.862181:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830741.862185:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830741.862193:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671281049856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830741.862200:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830741.862203:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830741.862205:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830741.862207:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5291 00000100:00080000:0.0:1787830741.862218:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830741.862222:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830741.862226:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830741.862236:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671281049984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830741.862237:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830741.862248:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830741.862250:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830741.862252:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830741.862255:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5291 00000100:00080000:0.0:1787830741.862257:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830741.862261:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830741.862274:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830741.862284:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671281050112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830741.862288:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830741/real 1787830741] req@ffff8ab3f075f0c0 x1874671281049600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830741.862303:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671281049600/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830741.862313:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830741.862317:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830741.862321:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281049600:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830741.862453:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281049728:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830741.862489:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830741.862826:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830741/real 1787830741] req@ffff8ab3f075f840 x1874671281049728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830741.862839:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671281049728/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830741.862849:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830741.862852:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830741.862855:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281049728:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830741.862862:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281049856:192.168.203.124@tcp:38:kworker.0 00000800:00000100:2.0:1787830741.862928:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830741.862942:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830741/real 1787830741] req@ffff8ab3f075ed00 x1874671281049856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830741.862962:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671281049856/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830741.862970:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830741.862973:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830741.862976:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281049856:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830741.862982:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281049984:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830741.863007:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830741.863018:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830741/real 1787830741] req@ffff8ab3f075f480 x1874671281049984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830741.863030:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671281049984/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830741.863040:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830741.863043:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830741.863046:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281049984:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830741.863052:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281050112:192.168.203.124@tcp:8:kworker.0 00000800:00000100:2.0:1787830741.863068:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:2.0:1787830741.863079:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830741/real 1787830741] req@ffff8ab3f075e580 x1874671281050112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830741.863090:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671281050112/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830741.863105:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830741.863108:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830741.863111:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281050112:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830742.886153:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830743.910199:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830744.934060:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830745.958070:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830746.982239:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830746.982260:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830746.982272:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830746.982276:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830746.982298:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5296 00000100:00080000:0.0:1787830746.982308:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830746.982316:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830746.982355:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671281050240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830746.982392:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830746.982396:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830746.982397:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830746.982401:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5296 00000100:00080000:0.0:1787830746.982405:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830746.982409:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830746.982418:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830746.982431:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671281050368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830746.982441:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830746.982444:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830746.982446:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830746.982450:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5296 00000100:00080000:0.0:1787830746.982454:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830746.982458:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830746.982475:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830746.982487:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671281050496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830746.982497:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830746.982500:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830746.982501:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830746.982504:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5296 00000100:00080000:0.0:1787830746.982507:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830746.982512:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830746.982516:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830746.982526:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671281050624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830746.982535:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830746.982539:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830746.982541:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830746.982545:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5296 00000100:00080000:0.0:1787830746.982548:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830746.982552:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830746.982556:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830746.982566:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671281050752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830746.983663:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281050240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830746.983776:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281050368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830746.983813:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281050496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830746.983833:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281050624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830746.983854:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281050752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830748.006836:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830749.031325:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787830750.054347:0:10945:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.124:988 00000400:00000100:3.0:1787830750.054404:0:10945:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.124@tcp at host 192.168.203.124 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787830750.054419:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830750.054463:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830750.054472:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830750.054479:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000800:00000100:3.0:1787830750.054486:0:10945:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.124@tcp 00000100:00100000:2.0:1787830750.054823:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830746/real 1787830750] req@ffff8ab3f075da40 x1874671281050240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830750.054846:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671281050240/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830750.054856:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830750.054860:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830750.054864:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281050240:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830750.054870:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830746/real 1787830750] req@ffff8ab3f075e580 x1874671281050368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830750.054881:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671281050368/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830750.054936:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830750.054940:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830750.054944:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281050368:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830750.054951:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830746/real 1787830750] req@ffff8ab3f075f480 x1874671281050496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830750.054963:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671281050496/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830750.054971:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830750.054975:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830750.054978:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281050496:192.168.203.124@tcp:38:kworker.0 00000100:00100000:2.0:1787830750.054985:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830746/real 1787830750] req@ffff8ab3f075ed00 x1874671281050624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830750.054999:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671281050624/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1787830750.055007:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830750.055009:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830750.055014:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830750.055018:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281050624:192.168.203.124@tcp:8:kworker.0 00000100:00100000:2.0:1787830750.055025:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830746/real 1787830750] req@ffff8ab3f075f840 x1874671281050752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830750.055035:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671281050752/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830750.055043:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830750.055046:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:2.0:1787830750.055049:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281050752:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830751.078542:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830752.102221:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830752.102234:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830752.102239:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830752.102251:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5301 00000100:00080000:2.0:1787830752.102263:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1787830752.102267:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830752.102281:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830752.102327:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b480 x1874671281050880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830752.103149:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830752.103153:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830752.103155:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830752.103161:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5301 00000100:00080000:2.0:1787830752.103166:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830752.103171:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830752.103184:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830752.103212:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439a40 x1874671281051008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830752.103221:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830752.103225:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830752.103227:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830752.103231:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5301 00000100:00080000:2.0:1787830752.103234:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830752.103238:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830752.103242:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830752.103252:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a940 x1874671281051136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830752.103260:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830752.103263:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830752.103264:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830752.103268:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5301 00000100:00080000:2.0:1787830752.103281:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830752.103286:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830752.103291:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830752.103302:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b840 x1874671281051264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830752.103311:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830752.103315:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830752.103317:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830752.103321:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5301 00000100:00080000:2.0:1787830752.103325:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830752.103329:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830752.103333:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830752.103352:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439e00 x1874671281051392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830752.103826:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281050880:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830752.103928:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830752.103997:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830752/real 1787830752] req@ffff8ab3f043b480 x1874671281050880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830752.104015:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b480 x1874671281050880/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830752.104026:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830752.104030:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830752.104034:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281050880:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830752.104042:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281051008:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830752.104064:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830752.104076:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830752/real 1787830752] req@ffff8ab3f0439a40 x1874671281051008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830752.104088:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439a40 x1874671281051008/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830752.104096:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830752.104099:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830752.104103:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281051008:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830752.104109:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281051136:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830752.104128:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830752.104139:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830752/real 1787830752] req@ffff8ab3f043a940 x1874671281051136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830752.104150:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a940 x1874671281051136/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830752.104157:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830752.104160:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830752.104163:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281051136:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830752.104168:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281051264:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830752.104186:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830752.104196:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830752/real 1787830752] req@ffff8ab3f043b840 x1874671281051264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830752.104206:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b840 x1874671281051264/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830752.104216:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830752.104219:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830752.104222:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281051264:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830752.104228:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281051392:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830752.104245:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830752.104258:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830752/real 1787830752] req@ffff8ab3f0439e00 x1874671281051392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830752.104267:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439e00 x1874671281051392/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830752.104287:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830752.104290:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830752.104293:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281051392:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830753.126309:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830754.150318:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830755.174146:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830756.198128:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830757.222041:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830757.222109:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830757.222115:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830757.222119:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830757.222129:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5306 00000100:00080000:0.0:1787830757.222139:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830757.222145:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830757.222193:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671281051520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830757.222221:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830757.222224:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830757.222226:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830757.222229:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5306 00000100:00080000:0.0:1787830757.222233:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830757.222237:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830757.222246:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830757.222258:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671281051648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830757.222284:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830757.222287:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830757.222288:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830757.222292:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5306 00000100:00080000:0.0:1787830757.222295:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830757.222555:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830757.222560:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830757.222574:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671281051776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830757.222584:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830757.222587:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830757.222589:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830757.222593:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5306 00000100:00080000:0.0:1787830757.222597:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830757.222601:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830757.222605:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830757.222614:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671281051904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830757.222622:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830757.222625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830757.222627:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830757.222630:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5306 00000100:00080000:0.0:1787830757.222634:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830757.222638:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830757.222641:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830757.222651:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671281052032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830757.223367:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281051520:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830757.223503:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830757.223609:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830757/real 1787830757] req@ffff8ab3f075f0c0 x1874671281051520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830757.223627:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671281051520/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830757.223637:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830757.223640:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830757.223644:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281051520:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830757.223651:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281051648:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830757.223673:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830757.223685:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830757/real 1787830757] req@ffff8ab3f075cb40 x1874671281051648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830757.223696:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671281051648/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830757.223703:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830757.223706:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830757.223709:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281051648:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830757.223715:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281051776:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830757.223733:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830757.223743:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830757/real 1787830757] req@ffff8ab3f075c000 x1874671281051776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830757.223753:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671281051776/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830757.223760:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830757.223763:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830757.223766:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281051776:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830757.223772:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281051904:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830757.223788:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830757.223798:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830757/real 1787830757] req@ffff8ab3f075de00 x1874671281051904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830757.223807:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671281051904/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830757.223817:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830757.223820:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830757.223823:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281051904:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830757.223828:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281052032:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830757.223852:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830757.223862:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830757/real 1787830757] req@ffff8ab3f075e940 x1874671281052032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830757.223873:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671281052032/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830757.223882:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830757.223908:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830757.223911:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281052032:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830758.247666:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830759.271147:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830760.294949:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830761.318062:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830762.342043:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830762.342056:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830762.342061:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830762.342074:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5311 00010000:00010000:1.0:1787830762.342076:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830762.342087:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830762.342099:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830762.342150:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cf00 x1874671281052160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830762.342180:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830762.342184:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830762.342186:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830762.342189:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5311 00000100:00080000:0.0:1787830762.342193:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830762.342198:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830762.342210:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830762.342226:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e940 x1874671281052288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830762.342231:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281052160:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830762.342250:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830762.342258:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830762.342259:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830762.342265:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5311 00000100:00080000:0.0:1787830762.342270:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830762.342283:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830762.342290:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830762.342305:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075de00 x1874671281052416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830762.342317:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787830762.342325:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:0.0:1787830762.342328:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830762.342332:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830762.342339:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5311 00000100:00080000:0.0:1787830762.342345:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830762.342351:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830762.342365:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830762.342381:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075c000 x1874671281052544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830762.342391:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830762.342397:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830762/real 1787830762] req@ffff8ab3f075cf00 x1874671281052160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830762.342414:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830762.342419:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830762.342429:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5311 00000100:00080000:0.0:1787830762.342434:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830762.342439:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cf00 x1874671281052160/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830762.342453:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00080000:1.0:1787830762.342455:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787830762.342464:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00080000:1.0:1787830762.342466:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:0.0:1787830762.342482:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075cb40 x1874671281052672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830762.342484:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281052160:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830762.342531:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281052288:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830762.342560:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830762.342585:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830762/real 1787830762] req@ffff8ab3f075e940 x1874671281052288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830762.342600:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e940 x1874671281052288/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830762.342608:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830762.342612:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830762.342617:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281052288:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830762.342624:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281052416:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830762.342646:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830762.342657:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830762/real 1787830762] req@ffff8ab3f075de00 x1874671281052416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830762.342669:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075de00 x1874671281052416/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830762.342678:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830762.342682:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830762.342686:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281052416:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830762.342692:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281052544:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830762.342712:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830762.342736:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830762/real 1787830762] req@ffff8ab3f075c000 x1874671281052544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830762.342749:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075c000 x1874671281052544/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830762.342761:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830762.342765:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830762.342768:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281052544:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830762.342774:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281052672:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830762.342793:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830762.342803:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830762/real 1787830762] req@ffff8ab3f075cb40 x1874671281052672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830762.342813:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075cb40 x1874671281052672/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830762.342821:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830762.342825:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830762.342828:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281052672:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830763.366068:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830764.392004:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830765.414507:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830766.438048:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830767.462018:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830767.462031:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830767.462036:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830767.462049:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5317 00000100:00080000:0.0:1787830767.462062:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1787830767.462062:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830767.462073:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830767.462122:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f0c0 x1874671281052800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830767.462152:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830767.462155:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830767.462157:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830767.462161:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5317 00000100:00080000:0.0:1787830767.462164:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830767.462168:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830767.462179:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830767.462191:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671281052928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830767.462201:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830767.462204:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830767.462206:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830767.462210:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5317 00000100:00080000:0.0:1787830767.462214:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830767.462218:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830767.462222:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830767.462231:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671281053056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830767.462240:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830767.462243:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830767.462245:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830767.462249:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5317 00000100:00080000:0.0:1787830767.462251:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830767.462256:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830767.462261:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:0.0:1787830767.462271:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671281053184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830767.462282:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830767.462286:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281052800:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830767.462287:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830767.462289:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830767.462297:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5317 00000100:00080000:0.0:1787830767.462301:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830767.462305:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830767.462309:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830767.462331:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671281053312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830767.462380:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830767.462434:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830767/real 1787830767] req@ffff8ab3f075f0c0 x1874671281052800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830767.462453:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f0c0 x1874671281052800/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830767.462463:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830767.462467:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830767.462471:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281052800:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830767.463186:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281052928:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830767.463224:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830767.463244:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830767/real 1787830767] req@ffff8ab3f075f840 x1874671281052928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830767.463259:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671281052928/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830767.463269:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830767.463273:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830767.463276:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281052928:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830767.463283:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281053056:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830767.463305:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830767.463317:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830767/real 1787830767] req@ffff8ab3f075ed00 x1874671281053056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830767.463349:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671281053056/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830767.463359:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830767.463363:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830767.463365:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281053056:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830767.463398:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281053184:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830767.463421:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830767.463433:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830767/real 1787830767] req@ffff8ab3f075f480 x1874671281053184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830767.463445:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671281053184/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830767.463455:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830767.463458:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830767.463461:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281053184:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830767.463467:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281053312:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830767.464427:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830767.464450:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830767/real 1787830767] req@ffff8ab3f075e580 x1874671281053312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830767.464470:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671281053312/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830767.464490:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830767.464494:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830767.464497:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281053312:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830768.486407:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830769.510367:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830770.534062:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830771.558134:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830772.582073:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830772.582241:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830772.582252:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830772.582256:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830772.582267:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5322 00000100:00080000:0.0:1787830772.582277:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830772.582295:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:0.0:1787830772.582334:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075da40 x1874671281053440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830772.582366:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830772.582370:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830772.582372:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830772.582376:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5322 00000100:00080000:0.0:1787830772.582380:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830772.582386:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830772.582398:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830772.582415:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075e580 x1874671281053568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830772.582430:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281053440:192.168.203.124@tcp:250:kworker.0 00000100:00080000:0.0:1787830772.582433:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830772.582448:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830772.582450:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830772.582456:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5322 00000100:00080000:0.0:1787830772.582460:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830772.582464:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830772.582471:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830772.582486:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f480 x1874671281053696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830772.582497:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830772.582501:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830772.582503:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830772.582508:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5322 00000100:00080000:0.0:1787830772.582520:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830772.582526:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830772.582532:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000800:00000100:1.0:1787830772.582538:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:0.0:1787830772.582551:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075ed00 x1874671281053824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830772.582563:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830772.582567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830772.582569:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830772.582573:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5322 00000100:00080000:0.0:1787830772.582579:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830772.582584:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:0.0:1787830772.582591:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:0.0:1787830772.582607:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f075f840 x1874671281053952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830772.582608:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830772/real 1787830772] req@ffff8ab3f075da40 x1874671281053440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830772.582686:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075da40 x1874671281053440/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830772.582701:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830772.582707:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830772.582712:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281053440:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830772.582742:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281053568:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830772.582766:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830772.582779:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830772/real 1787830772] req@ffff8ab3f075e580 x1874671281053568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830772.582792:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075e580 x1874671281053568/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830772.582800:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830772.582804:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830772.582808:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281053568:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830772.582815:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281053696:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830772.582832:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830772.582843:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830772/real 1787830772] req@ffff8ab3f075f480 x1874671281053696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830772.582854:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f480 x1874671281053696/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830772.582863:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830772.582866:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830772.582869:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281053696:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830772.582876:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281053824:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830772.582917:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830772.582928:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830772/real 1787830772] req@ffff8ab3f075ed00 x1874671281053824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830772.582940:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075ed00 x1874671281053824/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830772.582957:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830772.582961:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830772.582964:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281053824:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830772.582970:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281053952:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830772.582987:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830772.582997:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830772/real 1787830772] req@ffff8ab3f075f840 x1874671281053952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830772.583007:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f075f840 x1874671281053952/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830772.583018:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830772.583022:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830772.583025:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f075f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281053952:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830773.606133:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830774.630185:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830775.654042:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830776.679947:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830777.702131:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830777.702131:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830777.702140:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830777.702144:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830777.702153:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5327 00000100:00080000:3.0:1787830777.702162:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830777.702169:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830777.702203:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a421c0 x1874671281054080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830777.702224:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830777.702227:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830777.702229:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830777.702233:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5327 00000100:00080000:3.0:1787830777.702236:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830777.702240:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830777.702258:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830777.702268:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281054080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:3.0:1787830777.702272:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a430c0 x1874671281054208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830777.702284:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830777.702287:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830777.702289:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830777.702293:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5327 00000100:00080000:3.0:1787830777.702296:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830777.702300:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830777.702304:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830777.702314:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a41e00 x1874671281054336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830777.702323:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830777.702326:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830777.702328:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787830777.702337:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:3.0:1787830777.702338:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5327 00000100:00080000:3.0:1787830777.702343:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830777.702347:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830777.702352:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830777.702363:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281054464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830777.702372:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830777.702375:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830777.702377:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830777.702381:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5327 00000100:00100000:1.0:1787830777.702385:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830777/real 1787830777] req@ffff8ab3d0a421c0 x1874671281054080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830777.702386:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830777.702393:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830777.702401:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:1.0:1787830777.702413:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a421c0 x1874671281054080/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830777.702414:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281054592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830777.702446:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830777.702451:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830777.702455:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281054080:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830777.702477:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281054208:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830777.702495:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830777.702505:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830777/real 1787830777] req@ffff8ab3d0a430c0 x1874671281054208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830777.702516:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a430c0 x1874671281054208/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830777.702522:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830777.702524:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830777.702532:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281054208:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830777.702537:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281054336:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830777.702550:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830777.702571:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830777/real 1787830777] req@ffff8ab3d0a41e00 x1874671281054336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830777.702581:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a41e00 x1874671281054336/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830777.702589:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830777.702591:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830777.702594:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a41e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281054336:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830777.702600:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281054464:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830777.702617:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830777.702627:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830777/real 1787830777] req@ffff8ab3d0a43480 x1874671281054464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830777.702637:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281054464/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830777.702645:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830777.702648:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830777.702651:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281054464:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830777.702656:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281054592:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830777.702670:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830777.702679:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830777/real 1787830777] req@ffff8ab3d0a43840 x1874671281054592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830777.702689:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281054592/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830777.702698:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830777.702700:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830777.702703:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281054592:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830778.726079:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830779.750067:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830780.774068:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830781.798086:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830782.822052:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830782.822053:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830782.822062:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830782.822066:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830782.822077:0:15659:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5332 00000100:00080000:2.0:1787830782.822087:0:15659:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830782.822094:0:15659:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830782.822128:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a1c0 x1874671281054720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830782.822153:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830782.822156:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830782.822158:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830782.822161:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5332 00000100:00080000:2.0:1787830782.822165:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830782.822169:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830782.822179:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830782.822193:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671281054848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830782.822202:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830782.822206:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830782.822207:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830782.822222:0:15659:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5332 00000100:00080000:2.0:1787830782.822226:0:15659:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830782.822230:0:15659:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830782.822235:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830782.822245:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671281054976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830782.822255:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830782.822258:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830782.822260:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830782.822263:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5332 00000100:00080000:2.0:1787830782.822266:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830782.822269:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830782.822273:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830782.822283:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671281055104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830782.822292:0:15659:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830782.822296:0:15659:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830782.822297:0:15659:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830782.822302:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281054720:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787830782.822304:0:15659:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5332 00000100:00080000:2.0:1787830782.822312:0:15659:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830782.822316:0:15659:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830782.822320:0:15659:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830782.822332:0:15659:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671281055232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830782.822370:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830782.822414:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830782/real 1787830782] req@ffff8ab3f043a1c0 x1874671281054720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830782.822430:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a1c0 x1874671281054720/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830782.822440:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830782.822444:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830782.822448:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281054720:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830782.822455:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281054848:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830782.822475:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830782.822488:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830782/real 1787830782] req@ffff8ab3f043bc00 x1874671281054848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830782.822500:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671281054848/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830782.822508:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830782.822511:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830782.822514:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281054848:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830782.822520:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281054976:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830782.822540:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830782.822551:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830782/real 1787830782] req@ffff8ab3f04383c0 x1874671281054976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830782.822562:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671281054976/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830782.822569:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830782.822572:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830782.822574:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281054976:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830782.823710:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281055104:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830782.823749:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830782.823767:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830782/real 1787830782] req@ffff8ab3f043ad00 x1874671281055104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830782.823780:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671281055104/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830782.823791:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830782.823795:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830782.823798:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281055104:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830782.823805:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281055232:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830782.823823:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830782.823834:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830782/real 1787830782] req@ffff8ab3f04392c0 x1874671281055232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830782.823850:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671281055232/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830782.823860:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830782.823864:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830782.823867:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281055232:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830783.846093:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830784.870176:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830785.894034:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830786.918046:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830787.942023:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830787.942034:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830787.942039:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830787.942049:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5337 00010000:00010000:1.0:1787830787.942050:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830787.942059:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830787.942069:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830787.942107:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671281055360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830787.942132:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830787.942136:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830787.942138:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830787.942141:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5337 00000100:00080000:2.0:1787830787.942144:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830787.942149:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830787.942158:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830787.942168:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281055360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:2.0:1787830787.942173:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671281055488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830787.942198:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830787.942202:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830787.942204:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830787.942209:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5337 00000100:00080000:2.0:1787830787.942212:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830787.942217:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830787.942222:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830787.942235:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671281055616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830787.942245:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787830787.942251:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830787.942252:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830787.942256:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830787.942261:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5337 00000100:00080000:2.0:1787830787.942265:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830787.942270:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830787.942275:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830787.942287:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671281055744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830787.942294:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830787.942297:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830787.942299:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830787.942302:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5337 00000100:00080000:2.0:1787830787.942305:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830787.942308:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830787.942312:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830787.942320:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671281055872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830787.942461:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830787/real 1787830787] req@ffff8ab3f0439680 x1874671281055360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830787.942480:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671281055360/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830787.942490:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830787.942494:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830787.942498:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281055360:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830787.942520:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281055488:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830787.942543:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830787.942555:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830787/real 1787830787] req@ffff8ab3f043b0c0 x1874671281055488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830787.942568:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671281055488/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830787.942576:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830787.942579:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830787.942582:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281055488:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830787.942596:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281055616:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830787.942618:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830787.942629:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830787/real 1787830787] req@ffff8ab3f043a580 x1874671281055616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830787.942639:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671281055616/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830787.942646:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830787.942649:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830787.942652:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281055616:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830787.942658:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281055744:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830787.942676:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830787.942686:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830787/real 1787830787] req@ffff8ab3f0438f00 x1874671281055744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830787.942697:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671281055744/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830787.942710:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830787.942713:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830787.942716:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281055744:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830787.942723:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281055872:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830787.942741:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830787.942751:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830787/real 1787830787] req@ffff8ab3f0438b40 x1874671281055872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830787.942761:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671281055872/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830787.942769:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830787.942772:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830787.942775:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281055872:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830788.966107:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830789.990076:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830791.014340:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830792.038342:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830793.062051:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830793.062052:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830793.062062:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830793.062066:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830793.062076:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5342 00000100:00080000:3.0:1787830793.062086:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830793.062092:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:3.0:1787830793.062127:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a412c0 x1874671281056000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830793.062164:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830793.062168:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830793.062170:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830793.062173:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5342 00000100:00080000:3.0:1787830793.062177:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830793.062181:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830793.062190:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830793.062211:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a40000 x1874671281056128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830793.062219:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830793.062222:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830793.062224:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830793.062227:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5342 00000100:00080000:3.0:1787830793.062230:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830793.062234:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830793.062238:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830793.062246:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43c00 x1874671281056256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830793.062260:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830793.062263:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830793.062264:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830793.062267:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5342 00000100:00080000:3.0:1787830793.062271:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830793.062274:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830793.062278:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:3.0:1787830793.062288:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43840 x1874671281056384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830793.062306:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830793.062310:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830793.062312:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830793.062316:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5342 00000100:00080000:3.0:1787830793.062319:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830793.062323:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:3.0:1787830793.062327:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:3.0:1787830793.062338:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3d0a43480 x1874671281056512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830793.062405:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281056000:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830793.062462:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830793.062562:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830793/real 1787830793] req@ffff8ab3d0a412c0 x1874671281056000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830793.062582:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a412c0 x1874671281056000/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830793.062593:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830793.062598:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830793.062602:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281056000:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830793.062609:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281056128:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830793.062631:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830793.062643:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830793/real 1787830793] req@ffff8ab3d0a40000 x1874671281056128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830793.062655:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a40000 x1874671281056128/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830793.062663:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830793.062668:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830793.062671:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a40000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281056128:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830793.062677:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281056256:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830793.062698:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830793.062711:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830793/real 1787830793] req@ffff8ab3d0a43c00 x1874671281056256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830793.062723:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43c00 x1874671281056256/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830793.062731:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830793.062735:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830793.062738:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281056256:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830793.062745:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281056384:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830793.062765:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830793.062777:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830793/real 1787830793] req@ffff8ab3d0a43840 x1874671281056384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830793.062789:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43840 x1874671281056384/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830793.062799:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830793.062801:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830793.062804:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281056384:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830793.062811:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281056512:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830793.062832:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830793.062845:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830793/real 1787830793] req@ffff8ab3d0a43480 x1874671281056512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830793.062855:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3d0a43480 x1874671281056512/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830793.062864:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830793.062867:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830793.062870:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3d0a43480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281056512:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830794.089069:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830795.110236:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830796.147326:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830797.158479:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830798.182058:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830798.182795:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830798.182813:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830798.182817:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830798.182835:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5347 00000100:00080000:2.0:1787830798.182847:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830798.182854:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830798.182934:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438780 x1874671281056640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830798.182961:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830798.182964:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830798.182966:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830798.182970:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5347 00000100:00080000:2.0:1787830798.182974:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830798.182977:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830798.182987:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830798.183003:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438b40 x1874671281056768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830798.183014:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830798.183018:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830798.183020:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830798.183028:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281056640:192.168.203.124@tcp:250:kworker.0 00000100:00080000:2.0:1787830798.183029:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5347 00000100:00080000:2.0:1787830798.183035:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830798.183046:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830798.183052:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830798.183069:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0438f00 x1874671281056896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830798.183080:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830798.183085:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830798.183087:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830798.183090:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5347 00000800:00000100:1.0:1787830798.183140:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00080000:2.0:1787830798.183381:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830798.183390:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830798.183396:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830798.183409:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043a580 x1874671281057024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830798.183419:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830798.183421:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830798.183423:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830798.183427:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5347 00000100:00080000:2.0:1787830798.183430:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830798.183434:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830798.183438:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830798.183446:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043b0c0 x1874671281057152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830798.194687:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830798/real 1787830798] req@ffff8ab3f0438780 x1874671281056640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830798.194721:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438780 x1874671281056640/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830798.194742:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830798.194749:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830798.194754:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281056640:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830798.194793:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281056768:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830798.196067:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830798.196099:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830798/real 1787830798] req@ffff8ab3f0438b40 x1874671281056768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830798.198270:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438b40 x1874671281056768/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830798.198284:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830798.198289:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830798.198293:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281056768:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830798.198305:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281056896:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830798.198364:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830798.198388:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830798/real 1787830798] req@ffff8ab3f0438f00 x1874671281056896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830798.198400:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0438f00 x1874671281056896/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830798.198407:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830798.198411:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830798.198414:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0438f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281056896:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830798.198422:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281057024:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830798.198444:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830798.198456:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830798/real 1787830798] req@ffff8ab3f043a580 x1874671281057024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830798.198467:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043a580 x1874671281057024/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830798.198478:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830798.198481:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830798.198484:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281057024:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830798.198491:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281057152:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830798.198511:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830798.198521:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830798/real 1787830798] req@ffff8ab3f043b0c0 x1874671281057152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830798.198538:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043b0c0 x1874671281057152/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830798.198548:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830798.198551:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830798.198554:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281057152:192.168.203.124@tcp:8:kworker.0 00010000:00010000:1.0:1787830799.206018:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830800.231136:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830801.254140:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787830802.279603:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787830803.302022:0:48572:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830803.303410:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830803.303424:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830803.303428:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830803.303438:0:37:0:(import.c:510:import_select_connection()) MGC192.168.203.124@tcp: connect to NID 192.168.203.124@tcp last attempt 5352 00000100:00080000:2.0:1787830803.303457:0:37:0:(import.c:589:import_select_connection()) MGC192.168.203.124@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830803.303464:0:37:0:(import.c:624:import_select_connection()) MGC192.168.203.124@tcp: import ffff8ab3c584f000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000100:00100000:2.0:1787830803.303500:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f0439680 x1874671281057280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830803.303531:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830803.303535:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830803.303537:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830803.303540:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5352 00000100:00080000:2.0:1787830803.303544:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830803.303548:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8ab3c584d000: import ffff8ab3c584e000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830803.303557:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830803.303568:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04392c0 x1874671281057408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830803.303578:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830803.303581:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830803.303582:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830803.303586:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5352 00000100:00080000:2.0:1787830803.303589:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830803.303593:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8ab3c584d000: import ffff8ab3c584e800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830803.303597:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683205 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830803.303608:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043ad00 x1874671281057536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830803.303617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830803.303620:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830803.303622:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830803.303625:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5352 00000100:00080000:2.0:1787830803.303628:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830803.303633:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8ab3c584d000: import ffff8ab3d2db0800 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830803.303638:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 4485120, lost: 0. 00000100:00100000:2.0:1787830803.303648:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f04383c0 x1874671281057664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830803.303656:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 27aedc06-3022-4c11-aafb-6bd27e69b836->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830803.303659:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830803.303660:0:37:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830803.303664:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: connect to NID 192.168.203.124@tcp last attempt 5352 00000100:00080000:2.0:1787830803.303667:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830803.303670:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8ab3c584d000: import ffff8ab3d2db4000 using connection 192.168.203.124@tcp/192.168.203.124@tcp 00000008:00100000:2.0:1787830803.303674:0:37:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683205 ocd_grant: 3379200, lost: 0. 00000100:00100000:2.0:1787830803.303683:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8ab3f043bc00 x1874671281057792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830803.305003:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281057280:192.168.203.124@tcp:250:kworker.0 00000800:00000100:1.0:1787830803.305063:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830803.305133:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830803/real 1787830803] req@ffff8ab3f0439680 x1874671281057280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830803.305150:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f0439680 x1874671281057280/t0(0) o250->MGC192.168.203.124@tcp@192.168.203.124@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830803.305160:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830803.305164:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830803.305168:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f0439680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f0ad7a44-896c-4c9b-a0c3-55736bdf3a8f:10953:1874671281057280:192.168.203.124@tcp:250:kworker.0 00000100:00100000:1.0:1787830803.305175:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281057408:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830803.305197:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830803.305210:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830803/real 1787830803] req@ffff8ab3f04392c0 x1874671281057408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830803.305229:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04392c0 x1874671281057408/t0(0) o38->lustre-MDT0000-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830803.305237:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830803.305241:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830803.305244:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281057408:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830803.305250:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281057536:192.168.203.124@tcp:38:kworker.0 00000800:00000100:1.0:1787830803.305271:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830803.305287:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830803/real 1787830803] req@ffff8ab3f043ad00 x1874671281057536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830803.305299:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043ad00 x1874671281057536/t0(0) o38->lustre-MDT0001-mdc-ffff8ab3c584d000@192.168.203.124@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830803.305308:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3c584e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830803.305311:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830803.305314:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281057536:192.168.203.124@tcp:38:kworker.0 00000100:00100000:1.0:1787830803.305320:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281057664:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830803.305340:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830803.305350:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830803/real 1787830803] req@ffff8ab3f04383c0 x1874671281057664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830803.305360:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f04383c0 x1874671281057664/t0(0) o8->lustre-OST0000-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830803.305375:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830803.305379:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830803.305382:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f04383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281057664:192.168.203.124@tcp:8:kworker.0 00000100:00100000:1.0:1787830803.305387:0:10953:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281057792:192.168.203.124@tcp:8:kworker.0 00000800:00000100:1.0:1787830803.305405:0:10953:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.124@tcp 00000100:00100000:1.0:1787830803.305416:0:10953:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830803/real 1787830803] req@ffff8ab3f043bc00 x1874671281057792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830803.305437:0:10953:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8ab3f043bc00 x1874671281057792/t0(0) o8->lustre-OST0001-osc-ffff8ab3c584d000@192.168.203.124@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830803.305446:0:10953:0:(import.c:65:import_set_state_nolock()) ffff8ab3d2db4000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830803.305449:0:10953:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.124@tcp failed (-110) 00000100:00100000:1.0:1787830803.305453:0:10953:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8ab3f043bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:27aedc06-3022-4c11-aafb-6bd27e69b836:10953:1874671281057792:192.168.203.124@tcp:8:kworker.0 00010000:00010000:0.0:1787830804.326127:0:48573:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 75039 lines, 75039 kept, 0 dropped, 0 bad.